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ABSTRACT 



This thesis explores some difficulties associated with the use of the 
cloze procedure, particularly in relation to the interpretation of an 
individual's score on a cloze test used to determine whether the 
material from which the test is taken is, or is not, suitable for his 
instructional or independent reading. 

A number of cloze methodological considerations are discussed in detail. 
The literature relating to the development of comparable cloze and 
multiple-choice criteria for passage performance is reviewed. 

The Pilot Study explores the possibility that any one cloze test from 
a passage of prose might be more or less difficult than any of the other 
possible cloze tests from the same passage. After establishing means of 
categorizing the deleted words, the six possible eveiy sixth word deleted 
clcze tests from two 300 word passages from two different books at 
Grade Six reading level were used. On the basis of the categorization 
of the words deleted in each of these tests, the 1 easiest 1 and 'hardest 8 
tests for each passage were predicted. These were then tested on 196 
Grade Six children ii ight schools. For both passages the mean score 
for the predicted 'easiest' passage was significantly higher than that 
for the predicted 'hardest' passage from the same passage. 

The main investigation attempts to establish (i) the characteristics of 
deleted words which influence the difficulty levels of cloze tests; 

(ii) a simple means of adjusting the obtained cloze scores to allow for 
the relative ease or difficulty of replacement of the deleted words; and 

(iii) an operationally detennined range of scores which could serve as a 
criterion to indicate whether material is suitable for an individual 
child's independent or unsupervised reading. 
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For these purposes 112 different 350 word cloze tests were developed 
(the seven possible eveiy seventh word deleted patterns of 16 different 
passages). The 5,600 words included in these tests were placed in the 
categories established in the Pilot Study. The tests were given to 112 
Grade Six children in four schools. 

The results indicate that for the children in this study the easiest 
words to replace were those that are one syllable long, and/or are 
1-2 letters long, and/or are in conmon word lists, and/or are articles, 
conjunctions, prepositions or pronouns. A regression analysis deter min ed 
two formulas for adjusting the obtained scores but it was decided that 
the gain achieved would not justify the work involved in using them. 
The mean and standard deviation gave an estimate of the scores that 
could be expected to be achieved by two-thirds of the children doing 
cloze tests from material suitable for tteir independent reading. It is 
suggested that there is likely to be considerable overlap between expected 
scores for the independent and instructional levels. 

Some limitations of the study are discussed. 



CHAPTER 1 



INTRODUCTION 

The post-Sputnik era has seen the development of pressures both inside 
and outside the education profession to improve all education. As a 
consequence a great deal of public money has been poured into educational 
research and curriculum development. At the same time there has been a 
change in the organizational patterns of schools in order to meet the needs 
of an increasing and divergent school population, as well as to meet the 
needs of changing philosophies of educational theoiy and practice. 

One major dete rm i n a n t of the changes that have occurred has been the 
concentration on the individual learner. Although it has been long accepted 
that individuals are different, and their capacities to learn are different, 
the gap between acceptance in principle and acceptance in practice took a 
long time to bridge , It is only in very recent years that any real attempt 
has been made to individualize school programs. Until recently primaiy 
teaching, particularly in the upper part of the school, was generally 
conceived of in terms of formal instruction with the whole class as the 
working unit. Curricula were relatively clearly delineated and prescribed, 
and trainee teachers were taught specific methods to meet the demands of 
these curricula. 

The last decade has seen significant changes. Substantial modifications 
have been made to the primary school curriculum, leading to a move away from 
detailed prescription of content to the development of source materials 
(Warry and Fitzgerald, 1969). These changes have been characterized by a 
greater emphasis on higher cognitive and affective objectives (Ainley, 1972a). 
Thus, for example, the new mathematics courses emphasize themes and concepts 
rather than procedural drill, and the new science courses tend to emphasize 



process rather than content, with 'discovery learning 1 playing an important 
part (See Ainley, 1972b p. 29-30). 

Concomitant with this change in curricula has been an increasing 
emphasis on individual and small group instruction, a demand for flexible 
approaches and for the appropriate conditions' for learning (Grace, 1967). 
In fact in Victoria the traditional classroom with its associated teacher 

behaviours is no longer officially acceptable - »» the self contained 

classroom and the self contained school are obsolete." (Education Gazette 
and Teachers Aid, 1972, p. 513) 

Thus we are now faced with a variety of alternative classroom 
organizations, ranging from tentative variations on the old traditional 
theme to open classrooms (of infinite variety, philosophy, definition and 
effectiveness), family grouping and other multi-grading formats, together 
with an emphasis upon individualized procedures and increasingly open and 
flexible curricula. 

The changes in classroom organization, the 'opening 1 of courses and the 
emphasis on individualization of instruction bring with them a number of 
problems for the teacher, not the least of which is the need, more than ever 
before, to ensure that the instructional materials used by the child are 
suitable for his individual stage of development and ability. 

This is, of course, not a new problem. It is one that has always, in 
theory, existed. It is a problem, however, that has been intensified by 
modern educational practice, by the needs of the individual and, in some 
educational systems, by the demands for accountability and responsibility 
in educational practice. The more we move away from teacher centred, group 
teaching situations, to independent, individualized learning experiences, 
the more apparent is the need to ensure that the child can read and 
effectively cope with the materials provided for him. 

There are, therefore, a number of reasons why the teacher needs to 
have the means of determining the suitability of the material in any given 
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learning task for the individual. Among these reasons are: 

(a) the need to be able to monitor the student's learning during 
instruction so that instructional procedures and materials 
can be altered as needed; 

(b) the need to provide materia?. s that are difficult enough 
to challenge but sufficiently easy to ensure success; 

(c) the need to decide when a student has gained sufficient 
mastery of the content to warrant advancing him to a 
more complex unit; 

(a) the need to avoid children being given tasks that are too 
difficult or too easy and thus running the risk of them 
unnecessarily wasting time, becoming frustrated or 
anxious, or developing negative attitudes to self and 
learning. 



Determining suitability of materials. 

There are a number of ways in which the suitability of reading material 
for the individual might be determined. 

1 • Teacher and librarian estimates . 

Generally estimates of the suitability of materials made by teachers 
and librarians are subjective and as such are often open to a great deal of 
question. Klare (1963, p.81 ) states that "they are recognized as subject 
to considerable error", whilst Russell and Merrill (1951 ), in a study in 
which children's librarians rated the difficulty of well known juvenile 
books, found that such "expert" opinions do not show much general agreement. 

2. Readability formulas . 

This has been a common method of determining suitability of written 
materials. For this purpose the teim •readability 1 refers to the difficulty 
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level or comprehensibility of written prose. 

Readability formulas attempt to predict the likelihood of a given 
reading selection being understood by an individual or group of individuals. 
This is done by attempting to label selections of prose in texms of 
appropriate grade levels. There are a large number of elements involved in 
the concept of readability and Pry (1972, p. 204) makes one of the many 
attempts to enumerate them. Anderson (1 967) and KLare (1 963) give analyses 
of the factors involved. 

In general readability formulas make use of regression equations and 
take into consideration variables such as sentence length, number of 
syllables and the number of difficult or unfamiliar words. 

Ball and Williamson (1973) claim that the formulas devised by FLesch 
(1943, 1944), Lorge (1944) and Dale and Chall (1948) are simple to 

apply, yield consistent differentiation of standard sets of passages and 
have been shown to agree with observations of children's reading 
performances." (p.14) As a result their Readability levels of Children's 
Literature (Williamson and Ball, 1973) is based on the use of the Dale-Chall 
formula. 

Whilst readability formulas have been quite widely used by seme 
educationists, librarians, publishers, and others in the field of 
communication, there are a number of critics of the usefulness of such 
measures. For example, Blair (1 971 ) believes that there are too many aspects 
involved in readability that are not included in these formulas, such as 
contextual difficulty, abstractness of ideas, density of ideas, interest of 
subject, style appeal, material organization, size of type, type of ink, 
etc., etc. Otto and Smith (1 971 ) believe that mechanical formulas such as 
these work in opposition to any concept of readability which accepts the 
criterion that what an individual can read is, to him, readable. Thus, from 
their point of view, the only way to deteimine what is readable is by direct 
testing on the material by the individual. 
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Despite Ball and Williamson's (1 973) contention that readability 
formulas are simple to apply, it is probable that their mathematical nature 
limits their usefulness for many teachers and librarians. For example, the 
mathematical expressions of the three formulas they mention are as follows: 

(a) tlesch Reading Ease = 206.835 - .846 wl - 1.015 si. 

206.835 is a constant. 

wl = the number of syllables per 100 words. 

si = the average number of words per 
sentence. 

Reading Ease represents the grade level which would have to 
be attained in order to read the passage. 

(t) Lorge C 50 = 0.06 a + 9-55 b + 10.43 c + 1 .9892 

1 .9892 is a constant 

a = average sentence length. 

b = ratio of prepositional phrases to 
total number of words. 

c = ratio erf hard words (i.e. words not in 
Dale's 769 'easy words" List) to total 
number of words. 

C50 is the reading grade score of a pupil who answers one 
half of a series of test questions correctly. 

(c) Dale and Chall 

c 50 = -1579^ + .0496b + 3.6365 

3.6365 is a constant. 

a = average sentence length in words. 

b = percentage of words outside the Dale 
list erf 3,000 words. 

C50 is the reading grade score of a pupil who answers one 
half of a series of test questions correctly. 

Even a cursory glance at these formulas would indicate that, even if 



one discounts the problem of their mathematical nature, they are tedious and 
time consuming for the average classroom teacher to use for practical 
purposes, especially those that require the searching through of lengthy 
word lists to see if the words in the selected passages are in these lists, 
and the determination of prepositional phrases. 

There is also some contention about whether these formulas yield 
consistent differentiation* Carozzi (1972) points out that correlations 
between some of the formulas, e.g. ilesch and Dale and UJall, could be 
spuriously high as they include a sentence factor in common and have used 
the same criterion, viz. the McCall-Crabbs Test lessons. Michaelas and 
Tyler (in Eroese, 1 971 ) quote contradictory evidence regarding correlations 
between such formulas, whilst Blair (1 971 ) maintains that some show 
consistently higher scores (levels) than others, with the consequence that 
a readability level depends to a great extent on the measure used. Bormuth 
(1966) believes that the current formulas may hinder more than help because 
of their low predictive values and because they make poor guides for 
adjusting the difficulty of materials. 

Finally, although KLare (1 952) argues that "... readability formulas 

are sufficiently accurate for estimating the comparative readability 

of adult materials 11 (p.397) (my underlining) and Lorge (1948) points out 
that the readability index is an estimate and not intended as a precise 
indication, Carozzi (1972) indicates that "teachers and publishers tend to 
treat readability formulas as though they were precise measures." (p. 71 ) 
So although Spache and Chall each pointed out that levels from formulas are 
only accurate to within t 1 year of reading age (McLeod, 1962), the 
formulas have been used to make distinctions of 1 to 2 months in the 
reading difficulty of books without also mqlH rig the error involved quite 
clear. (See, for example, Bird and Ealk, 1 971 ) • 

3. Direct testing . 

An alternative to the subjectivity of teacher/librarian estimates and 
the problems associated with the use of readability formulas is that of 
testing the reading material on the child directly. 
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In most versions of this procedure the student is asked to read a 
passage that is thought to be representative of the book or instructional 
materials, and then answer some questions about the passage - the questions 
usually being of a multiple-choice foimat. 

It has been accepted for a long time (e.g. Kilgallon, 1942; Betts, 
1946) that if a child is able to answer at least 9(# of the questions based 
on the material he has read then the material is said to be at his 
independent level, and is therefore suitable for use in his unsupervised 
study and voluntary reading. If he is able to answer at least 75# of the 
questions, then the material is said to be at his instructional level , and 
suitable for use in his supervised instruction. If he is unable to answer 
at least 5<# of the questions then the materials are said to be too 
difficult, or unsuitable, or at his frustrational level . These levels, 
which have been operationally defined, have been used in readability 
formulas such as those of Dale-Chall, Lorge and ELesch, where the criteria 
has been based on either 50^ or 75# comprehension on the McCall-Crabbs Test 
lessons. 

The direct testing approach has been recommended in a number of reading 
textbooks, e.g. Bond and Tinker (1967), Della-Piana (1968), Harris (1962) 
and Russell and Thompson (1966). 

There is, however, a major problem associated with the use of direct 
testing of material on the individual child. That problem is the dependence 
on multiple-choice questions as the criterion for performance . The problem 
is accentuated by the fact, that for most practical purposes, it is the 
teacher himself who writes the multiple-choice questions, Wesman (1971 ) 

writes: "Item writing is essentially creative - it is an art (it) 

requires an uncommon combination of special abilities and is mastered only 
through extensive and critically supervised practice. » (p.81 ) It is 
probable that very few teachers are sufficiently trained in the skills 
necessary to construct test questions that can meet the criteria for even 
relatively loose standards of replicability. 
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The requirements for items to exhibit the necessaiy clarity and 
effectiveness are numerous. Wesman (1971 ) lists 12 general suggestions 
as well as another 1 2 specific suggestions for the writing of multiple- 
choice items* (See Appendix A) 

It is probable therefore that the following difficulties may be- 
associated with direct testing where the criterion is a test involving 
multiple-choice items devised by the classroom teacher: 

(a) It may be difficult to determine whether the answers given by the 
child reflect the difficulty of the passage (material), or the 
difficulty (lack of clarity) of the questions. 

(b) It may not be known how far the subjectivity or preferences 
(prejudices/beliefs/attitudes) of the test constructor affect the 
items and therefore the outcomes. 

(c) It may not be known if the questions set on any passage are 
sufficient in number to adequately sample the content of the 
passage or are sufficient in scope to be an unbiased sample of 
all the questions that could ha v e been asked. 

(d) As construction of these tests is time consuming it is most 
unlikely that the average classroom teacher will/can spend the 
time required to write carefully constructed items and expose 
them to expert editorial scrutiny (as suggested by Wesman, p.111). 

As a result it is possible that the difficulty, the reliability, and 
even the validity of such tests are likely to vary from any one teacher to 
another and from any one time to another. Hence, there is no certainty as 
to what a score of 75# or 9$ on these tests might really mean - no 
certainty as to whether they are accurately predicting frustrational, 
instructional or independent levels of reading. 

It is in this context of doubt that the cloze procedure (Taylor 1953) 
has been introduced as a viable solution to a measurement problem. This 
procedure involves the deletion of words from a passage of prose and the 
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measurement of the ability of the individual to replace these omitted words. 
(The procedure is reviewed in detail in Chapter 2) 

Because the cloze procedure asks no questions, involves no memoiy 
component, is constructed by the simple and objective mechanical deletion 
of words, and does not appear to be measuring a student's familiarity 

with the content of the passage" (Simons, 1971, p. 347), it has been seen by 
some researchers (e.g. Boimuth, 1967, 1968; Rankin and Culhane, 1969; and 
Anderson and Hunt, 1972) as a realistic alternative to the problems posed by 
the potentially inaccurate multiple-choice testing criteria for the 
measurement of passage performance. With the cloze procedure, it is claimed, 
it is possible to have the advantage of direct testing of the individuals 
ability to comprehend the material associated with the use of an accurate 
and objective measure of this comprehension. 

The purpose of this thesis is to investigate the effectiveness of the 
cloze procedure for this purpose. Anderson (1971a) claims it to be 
11 ... one of the most promising techniques to emerge in recent years for 
measuring comprehension and reading difficulty, (p. 181 ) whilst KLare 
(in Groff, 1971 ) believes it to be " ... clearly one of the most, if not 
the most, convenient and widely applicable techniques ever suggested for 
studying text. 11 (p.677) These claims need to be explored in the context 
of the measurement of passage performance. 

The next chapter will deal with the rationale and effectiveness of the 
cloze procedure, and will review its use by researchers afc a means of 
predicting the suitability of reading materials. 
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CHAPTER II 

THE CLOZE PROCEDURE AMD 
PARAGRAPH PERFORMANCE 

Although as recently as six years ago it could be written that the 
cloze procedure was familiar only to a small number of reading and 
language specialists (Spache, 1968), its use over recent years has 
developed greatly (see, e.g. the bibliographies of Bqyce, 1973a, and 
Klare, Sinaiko and Stolurow, 1972). Not only is the cloze being 
extensively used by researchers in reading and language, but its inclusion 
in some reading texts (e.g. Pry, 1972, and Strang, 1968) and the numerous 
articles explaining its practical usefulness for classroom teachers (e.g. 
Anderson, 1968; Bortnick and Lopardo, 1973; Culhane, 1970; Galloway, 1973; 
Guice, 1969; Humphreys and Kay, 1971; Mork, 1971; Oiler, 1972; Oiler and 
Conrad, 1971; and Weintraub, 1968) have made it a potential measuring 
tool for the practising classroom teacher* 

The procedure, which was introduced by Taylor ( 1 953 ) f involves the 
mutilation of passages of prose by the deletion of words on some mechanical 
basis. Introduced as a means of deteimining readability of material, it 
has been used for a wide variety of purposes over the years (see, e.g. 
Bickley, Ellington and Bickley, 1970, and Bqyce, 1973b, p. 34.) 

Rationale 

In introducing the cloze Taylor drew on Miller's (1 951 ) work in 
communication theory, Osgood's ( 1952) "dispositional mechanisms" and the 
principles of random sampling. He chose the name 'cloze' as a derivation 
from the Gestaltist law of closure - the principle that behaviour or mental 
processes tend towards completing or 'closing' as far as circumstances permit. 
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The procedure as introduced by Taylor was to systematically delete 
words in a passage of prose and evaluate the success the reader had in 
accurately supplying the missing words. He reasoned that if the individual 
could understand the message when words were deleted, and could replace the 
words exactly, he was experiencing a form of closure. In order to make 
these cloze responses the individual had to decide from the context that 
remained what the missing parts were. Therefore the reader was required 
to have an adequate grasp of the language structure on the page as well as 
a grasp of the basic tone and substance of the passage. Thus Taylor 
claimed that the procedure provides "... a measure of the aggregate 
influences of all factors which interact to affect the degree of corres- 
pondence between the language patterns of transmitter and receiver." 
(1953, p.432) 

Anderson (1971 a) maintains that there is little empirical evidence for 
the explanation of 'closing 1 broken language patterns in the same way as 
one 'closes' an incomplete circle. He proposes that a more defensible 
rationale lies in current communication theory. The deletion of words is 
seen as 'noise 1 and the reader's task is seen as that of reconstructing the 
la ng ua g e patterns by making the most likely replacement in the li^it of his 
language system and the grammatical and semantic cues that are available. 
(See Figure 1 ). 



Source System 



Message System 



Receiver System 



Writer or 
Speaker 



Printed or 
Spoken Words 



T 



Noise System 



Reader or 
Listener 



Mutilation 
of message. 



Figure 1 A Model for the Language Correspondence of a Source System 
to a Receiver System. (Anderson, 1971, p. 179) 
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However, although Taylor sees the replacement of missing words as 
•closing 1 , and Anderson sees it as eliminating 'noise 1 , these different 
interpretations appear to have no practical implications, 

Clark and Johnson ( 1 973 ) argue that on the basis of Taylor rationale 
"... the cloze procedure could produce spurious comprehension scores for 
poor readers since some words substituted will simply reflect automatic 
response to grammatical patterns rather than appreciation of the full 
meaning of the sentences or language units involved," (p.15) This will 
occur, they argue, because complete use of all contextual clues is not 
necessary for the replacement of 'functional 1 or 'structural 1 words (e.g. 
pronouns and prepositions), and these are easier to replace than 'content 1 
words such as nouns, adjectives and verbs. MacGinitie (1 966) also points 
out that missing words can often be restored correctly without "understanding" 
of the passage because all that is needed is a recognition of familiar 
patterns of expression. He feels that unless the blanks in the cloze test 
are appropriately selected, the cloze scores may be more a measure of 
language redundancy than of comprehension. This matter of the ease of 
replacement of various parts of speech is explored in the pilot study 
reported in Chapter 3 of this thesis. 

It should be noted that the cloze procedure is not the same as 
'fill-the-gap' or 'sentence-completion 1 exercises. Tirpically these 
exercises are used to gain a measure of a person's knowledge of specific 
and usually independent points of information, and therefore the deletions 
are chosen quite subjectively. On the other hand cloze procedures are 
mechanical and therefore objective, the concern being with a contextually 
related series of deletions rather than with isolated ones. 

Methodological considerations . 

Although consistently referred to as a simple procedure, a survey of 
the literature indicates that a wide variety of practices are used in the 
construction of tests as well as in the scoring. Taylor's ( 1 953 ) intro- 
duction was a completely mechanical procedure of choosing words to be 
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deleted on a random or every nth wor d basis, and calling for the exact 
replacement of the deleted words. However, subsequent developments have 
varied a number of different factors to the extent that it is difficult 
to talk about jhe cloze procedure, and which make it important that those 
who report research using the cloze indicate precisely what method they 
have used. The following sections attempt to discuss and clarify some of 
the variations that have occurred. 
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Frequency of word deletions . 

There are two commonly used word deletion approaches, viz., random 
deletion and nth wor a deletion, although the latter is far more common in 
the reported research. Amongst those who have used nth wor a deletion the 
deletions usually vary between every fifth and every tenth word. Culhane 
(1970) suggests that every tenth word should be used with textual materials 
laden with fact, but that a count as low as every fifth word may be used 
satisfactorily with narrative materials. Probably the greater majority of 
researchers use an every fifth word deletion on the basis that MacGinitie 
( 1 961 ) , in an investigation into contextual constraints in English prose 
paragraphs, found that the influence on word choice appears to decrease 
rapidly with distance of the context, and that after about five words 
distant the context has relatively little effect on the choice. Johnson 
(1968) and Anderson (1 969) come to the same basic conclusion. Kerr (1970) 
reports that Anderson (1969) and Kerr and Smith (1968) have found that an 
every eighth word deletion pattern worked successfully with Australian 
primary school children, although he does not elaborate on the statement. 
The present author has found, in unpublished and unreported investigations, 
that younger primary school children doing cloze exercises for the first 
time find the deletion of every fifth word a rather daunting experience. 

Most published investigations give no reason for the choice of every 
fifth, sixth, seventh, etc. word, and it would seem that in many cases the 
choice is purely an arbitrary one. In the cloze exercises devised for 
this investigation two different deletion patterns were used. In the Pilot 
Study (see Chapter 3) an every sixth word pattern was used. In the major 
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study an eveiy seventh word deletion pattern was used. These deletion 
patterns were chosen as a compromise between the Clark and Johnson (1972) 
and the Anderson (1969) eighth, both of which had been found satisfactoiy 
with Australian children, and the majority of other studies which use an 
eveiy fifth word deletion pattern. 

As Clark and Johnson (1973) say, "... a more rigorous and selective 
deletion system is warranted, if proper account of contextual constraint 
is to be taken and children's errors in replacing deletions are to have 
any practical significance in relation to particular passages." (p. 17). 
Certainly there needs to be research carried out to determine what are the 
most effective deletion systems according to the age of the person doing 
the exercise, and according to the type of content of the material. 

Clark and Johnson ( 1 973 ) also raise the issue of wildly fluctuating 
difficulty levels associated with a random deletion approach. The major 
purposes of the Pilot Study reported in Chapter 3 are to see if there are 
fluctuating difficulty levels with an nth wor a deletion pattern, and to 
see if, for example, an eveiy seventh word deletion pattern is used that 
there is no reason to believe a cloze exercize deleting the first word and 
then eveiy seventh will necessarily be of equivalent difficulty to one 
starting with the deletion of the second, or the third, or the fourth, etc. 
and then eveiy seventh word thereafter. 

Finally in this section on frequency of word deletions there is a 
related question that should be considered, viz. just what constitutes a 
deletion element. Jongsma ( 1 971 ) indicates that although researchers 
usually answer this on a logical basis, there is in fact no research 
evidence available for guidance. Thus, e.g., should numerals be subject 
to deletion and should hyphenated words be treated as single units or 
broken up into their separate parts? Klare, Sinaiko and Stolurow (1 972) 
state that a word is usually defined "by the white spaces separating it 
from other words (e.g. don't, U.S.A., 2, 182, and re-enter would all be 
single words). Commas, apostrophes, and hyphens should be deleted along 
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with the rest of the word." (p. 85) They believe that hyphenated words 
should be deleted as units only when one of their elements represents a 
bound rather than a free morpheme, as, for example, the co- in co-chainnan. 
In the cloze exercises used in this study numerals were included as units 
for deletion, and hyphenated words were broken up into their separate 
parts. 

Type of words deleted . 

Although introduced as an nth (any word) or random deletion procedure, 
a number of researchers have carried out investigations using specific 
word deletions. In his second study (1 957) Taylor used three types of 
word deletions: "any 1 words, 'hard 1 vrords (adverbs, verbs and nouns) and 
'easy' words (e.g. pronouns and articles). He found that for some purposes, 
e.g. measuring prior knowledge of technically worded material, the deletion 
of 'hard* words was the best measure. However, for most purposes he found 
the f any' word deletions were superior to the other forms of deletion. 
Greene (1965) modified the procedure by restricting words eligible for 
deletion to nouns, verbs, adverbs and adjectives. Louthan (1965), whilst 
using a purely mechanical form of deletion for part of his study, used a 
number of specific deletions such as proper and common nouns, as determined 
by morphology and syntax; specific verbs exclusive of function verbs; and 
specific modifier, adjective and adverb, all on a ten per cent deletion 
basis. He found that with all the classes listed above the lexical and 
grammatical redundancy was not great enough to bridge the gaps in the prose. 

Rankin (1958) refers to the any word deletion by mechanical n*h word 
as structural deletion, and by specific word type as lexical deletion. 
He assumes that passages comprising lexical deletions measure the 
understanding of substantive content, while structural deletions involve 
an landers tanding of the inter-relationship of ideas and are more highly 
influenced by intelligence. Although Jongsma ( 1 971 ) admits that there is 
some evidence for the psychological reality of this dichotomy, he also 
maintains that it is not as convincing as many would have us believe. He 
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believes that there is as yet insufficient evidence to suggest that the 
distinction applies equally well across all age and grade levels and 
across all "types of reading materials. 

Schlesinger (1968) is also critical of the structural-lexical concept. 
He believes that it does not take into account the deep versus surface 
structure of the sentence. Although the example he gives (1968, p.154) is 
rather extreme - he uses an every second word deletion - his point is worth 
considering, that instead of continuing to rely on the grammatical elements 
of the sentence, conventionally defined by establishing word classes or by 
using parts of speech and word categories, an attempt should be made to 
focus on the linguistic variable of word order or sentence structure. 

Ohnmacht, Weaver and Kohler (1970) explored the relationship between 
the cloze and closure in a factorial study. They used four types of 
deletion systems defined as follows: "structural 11 , "lexical 11 , "abstract 
nouns" and "concrete nouns". Eactor analysis identified a number of 
patterns which differentiated the tests. The cloze tasks could be broken 
into two dimensions: (a) the 'lexical 1 and 'abstract noun 1 deletions 
were more closely related to vocabulary and, (b) the 'structural 1 and 
'abstract' noun deletion forms separated out in another dimension. None 
of the closure tasks had a major relationship with the experimental cloze 
tests, and the latter showed a positive relationship with performance on 
the associational tasks. 

They then suggest that - 

'the fact that responses to cloze tasks reflecting essentially 
gross deletion strategies align themselves with crude measures 
of comprehension does little to throw light upon the funda- 
mental nature of comprehension other than to indicate that one 
can measure what passes for comprehension in more than one way. . .'(p. 21 5) 

and continue - 

•Rather than standardizing a particular cloze deletion type, 
exploration of a wide range of deletion types which are 
related to particular linguistic and psychological hypotheses 
is needed. ' (p.215) 
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Bowers and Nacke (1971-72) believe that the generative transformational 
theory of Chomsky (1957, 1965) means that one needs to re-appraise some of 
the use of the cloze procedure. They point out that although some 
researchers, e.g. Hankin (1959), Weaver (1965) and Treisman (1965), have 
modified the raw cloze procedure to allow for the differences between 
structural and referential morphemes, these attempts have not overcome the 
considerable problems generative theory presents for the theoretic basis 
of the cloze procedure. Bowers and Nacke present a tentative algorithm 
for the deletion of redundant words in the English language which they 
believe can form the basis of restitution tests "which will be both valid 
and illuminating" (p.31 ). As yet there has been no reported research using 
this algorithm. 

Despite the doubts recently cast by linguists, the cloze procedure 
continues to be used, mainly on the basis of an 'any word 1 deletion. 
Although Taylor himself used specific word deletions in his 1957 study he 
maintained that for readability purposes to "restrict deletions to particular 
kinds of words is to ignore the fact that those kinds of words may not occur 
equally often in different materials. The difference of frequency of 
occurrence may itself be a readability factor; if so, its effect should be 
included in - not excluded from - the results." (1957, p. 25) On the other 
hand Clark and Johnson (1973) suggest that this might be just as much an 
argument for carefully analysing the passages before determining the type 
of deletion. 

In the investigations carried out by the author for this thesis only 
mechanical 'any word 1 nth deletions were used - the cloze procedure as 
originated by Taylor (1953). However, the purpose of the Pilot Study 
reported in Chapter 3 was to investigate the extent to which different 
types of words may not occur equally often in different cloze versions of 
the same material. 

Scoring Cloze Tests 

It is usual for cloze tests to be scored for exact replacement of the 
deleted words, although various other methods have been explored. 
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For example, Guice (1 969) graded on the basis of two points for exact 
replacement and one point for a synonym. Weintraub (1968), although 
reporting that most research has been carried out using exact work replace- 
ment, suggests that synonym replacement is allowable. Miller and Coleman 
(1967) using two scoring methods, viz. (a) exact replacement, and 
(b) 3 points for exact replacement, 2 for a synonym and 1 for correct part 
of speech, found a correlation between these methods of 0.99. This, 
together with the evidence of research, e.g. Taylor (1953), Rankin (1 957), 
Ruddell (1964) and Bormuth (1964, 1965), suggests that not only is scoring 
for exact replacement simpler and more reliable (as no subjective assessments 
have to be made as to what are allowable alternative replacements), but that 
scoring allowing for synonym replacement does not lead to better discrimina- 
tion between individuals. 

On the other hand it could be reasonably argued that if the cloze test 
was being used to consider the individual's performance rather than to 
assess his performance relative to others, that some purpose might be 
achieved by scoring for synonyms and logical replacements. Schoelles (1 971 ) 
believes that when the procedure is being used for measuring student ability 
the scoring of synonyms is desirable. She argues, for example, that 
enriched vocabulary use - such as 'constructed 1 for 'made 1 - should not be 
penalized. Boyce (1972) in a study in which responses to a cloze test 
were scored for (a) exact replacement, and (b) exact replacement or synonym 
or logical replacement, found that the mean scores increased from 21.29 for 
scoring method (a) to 28.78 for scoring method (b). An investigation of 
the accepted synonyms and logical replacements indicated that, in some 
cases at least, the exact replacement word was not the common usage word 
of the children. 

Oiler (1972) argues that although mean scores tend to be higher when 
acceptable substitutes are allowed, the increase in total test variance is 
so small as to be scarcely worth the extra effort involved, and Bormuth 
(1965) suggests that exact word replacement is required for validity. 
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For practical purposes it is probably reasonable to maintain only the 
exact replacement scoring system as any other method loses objectivity, and 
the work involved in determining what are acceptable synonyms and logical 
replacements is considerable. One possible solution is the development of 
clozentropy (Darnell, 1970). Clozentropy was developed as a procedure for 
testing English language proficiency of foreign students. It has amongst 
its theoretical assumptions one that states "... that a measure of profi- 
ciency in language should index one's ability to conform to existing group 
norms of language rather than to some prescriptive model or idealized 
language pattern." (p. 36) Thus, although Darnell uses the cloze technique, 
he also uses an entropy measure which indexes the compatibility of an 
individuals responses with those of a selected criterion group. This 
leads to a scoring system that is mathematically precise, which avoids 
entirely the right/wrong judgements on an item by item basis, but which is 
rather complex. 

In the two studies carried out for the purpose of this thesis only 
exact replacement scoring was used, mainly because in both cases the results 
were being compared with, or related to, other studies using exact replace- 
ment scoring. 

Number of deletions . 

Kerr (1970) points out that because random or nth WO rd deletions lead 
to a number of non-discriminating items being included, the reliability of 
the test is lowered if there are only a few items. Thus the test has to be 
long enough to be reliable, but not long enough to cause fatigue and boredom. 
Taylor (1 956) suggested that 50 items led to a stable score, and this was 
supported in principle by Bormuth (1964). In a later study ( 1965b) Bomruth 
presented a table, based on Lord's ( 1 955 ) formula for standard errors, which 
allows an estimate of the standard error to be made according to the number 
of deletions and the number of subjects. 



Obviously the length of the test is affected by the rate of deletion. 
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Thus a fifty item every fifth word deletion exercise would be much shorter 
than a fifty item test with every eighth word deleted. Although Anderson 
( 1971b) suggests that it makes very little difference "except in terms of 
efficiency and reliability", (p.38) whether one uses every sixth, seventh, 
or eighth word, it does, of course affect the total length of the passage 
being used. There appears to be no consistent body of research to indicate 
what is the best deletion pattern to use according to the age of the child 
and the type of material, and hence there is a possibility that longer 
passages may be needed for younger children than with older ones. Related 
to this is the matter of motivation. There appears to be no research on 
how performance on the cloze affects the motivation of the child to continue. 
It could be hypothesized that the smaller the number of words between gaps 
(deletions), the more difficult it is for the younger child. 

In all the tests devised for use in this thesis a fifty item cloze was 

used. 

Related to the question of the length of tests, and a question rarely 
mentioned in discussions of the procedure, is that of whether a 'run-in' 
should be used before the deletions actually commence. There appears to be 
some confusion on this matter. Some researchers start deletions from the 
first sentence, others leave the first sentence or two, whilst others leave 
as much as the first paragraph of the material before commencing deletions. 
Oiler (1972) writes, "As is customary, the first and last sentences of each 
paragraph were left intact." (p.152). Klare, Sinaiko and Stolurow (1972) 
state that although some writers suggest that no words be deleted from the 
first and last sentences of a passage, they feel it to be unnecessary 
"... except for subjects like young children, near-illiterate adults, or 
such who need a great deal of help." (p.85) Anderson, (1972 - personal 
communication) writes, "Your suggestion of leaving an initial paragraph 
unmutilated is sound though depending on the length of the paragraph it 
may not be necessary to leave the whole paragraph ... but it is important, 
I agree, to set the scene so-to-speak. " 
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Another factor not usually mentioned, but of practical importance, is 
whether the subjects read through the material first before attempting to 
replace the deleted words. This, together with the 'run-in 1 factor, may 
have some influence on the strategy the subject uses, and therefore on 
his score. It is feasible that leaving the initial paragraph, or at least 
part of it, unmutilated, together with the instruction to read through the 
whole task first before attempting to fill in the gaps, would allow the 
subject to approach the task as a whole because he has a better grasp of 
the total context, mood, style, etc. On the other hand, if he simply 
starts at the beginning without any overview, replacing each omission as he 
comes to it, he may treat the passage as a series of sub- tasks, as a series 
of bits of information. If this does in fact happen it could account for 
some of the replacements which, although patently wrong in the total 
context of the passage, make sense in the context of the few words 
immediately preceding and immediately following the particular deletion. 

Anderson (1972 - personal conmunication) states that the usual 
instruction is to read through the whole passage and then fill in the 
missing words. KLare, Sinaiko and Stolurow (1972) and Bormuth (1964b) do 
not mention this in their sets of instructions, nor indeed does Anderson 
(1971 a). However, in another article, Anderson (1 971b) gives the following 
instructions for the use of the cloze with primary school children: 
" ... I want you to read each stoiy and guess the missing words. Then I 
want you to print in each space the one word you think should go there." 
(p.39) 

The majority of journal articles do not mention the instructions 
given. This would seem to be an unfortunate emission as it may have a 
major influence on the strategy/strategies used by the children to do the 
exercises. There, in fact, seems to have been veiy little research carried 
out into the strategies used by the subjects. Jenkinson (1957) selected 
high school students who had done vexy well, or poorly, on cloze tests and 
asked them to verbalize their reasons for the insertion of words on 
another test. These verbalizations were then analysed and showed that the 
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higher scoring students were much better in recognizing syntactical cues, 
sensitivity to style, language structure etc. Although there would be 
problems involved, it would seem feasible to ask children to introspect 
about the way they went about the task, to identify the strategies used, 
and then to test them experimentally by using vaiying forms of instructions. 

The instructions used in the testing carried out by the author for 
this thesis can be found on page 53. These instructions include the 
sample exercise given to acquaint the children with the procedure. All 
the cloze exercises devised included a 'run-in 1 before deletions commenced, 
the length of which varied from exercise to exercise. The actual length 
of the 'run-in 1 was determined to some extent by the fact that in the major 
investigation the exercises were photostats of the original text from the 
book and as far as possible the exercise was kept to one page only. 

Format of the exercise . 

General practice is for exercises to be compiled by typing out the 
passage and replacing the deleted words with a blank space of standard 
length, usually ten or fifteen typewriter spaces. The tests are then 
presented in duplicated foim with the subjects writing the replacement 
words in the spaces provided. Unless one has a see-through template with 
the correct replacements written on it, the correction of cloze exercises 
compiled in this manner can be very frustrating. An alternative is to 
number the spaces and have numbered blanks of standard length on the light 
hand margin of the page, or on a separate answer sheet. This method greatly 
facilitates correction as a simple vertical answer card can be used to 
match up answers. There may be one possible disadvantage however in that 
subjects have to search for the correct place to commence each time after 
having written the answer in another place and may thus lose the thread 
of the passage. 

Anderson (1971 a) has suggested another format. He claims that his 
research has shown that blanks of the same length as the deleted word are 

ERIC v53 



23. 



an effective alternative. He therefore suggests that cloze exercises can 
effectively be constructed by glueing paper over the words in the original 
that are to be deleted, and then photocopying the passage. Such a method 
would mean that the size of print, length of deleted word, illustrations, 
and page layout could be contextual cues involved in the exercise. 
Although KLare, Sinaiko and Stolurow (1972) claim that standard size 
blanks should be used and that use of blanks of the same size as the 
deleted words provides undesirable cues, it would seem reasonable to use 
whatever cues the materials can give. After all, what we are trying to 
determine is whether the child (subject) can comprehend the material - 
as it is in the book. 

The present author used a variation of Anderson's photostat format 
for the main investigation in this thesis. In all there were 112 different 
cloze exercises. To have produced these in typewritten duplicated foim 
would have been veiy costly. The cutting out or pasting over of words to 
be deleted turned out to be a veiy frustrating and time consuming task. 
Instead, words to be deleted were obliterated by the use of white liquid 
retype. Although Anderson suggests that the students can write in the 
answers in the spaces left in this photostat foxmat, this author found 
that the space left with many of the small type forms together with the 
general size of primary children's writing, made this impractical. Thus 
each of the whited-out blanks was numbered and a separate answer sheet 
.provided. (See Appendix H) For the pilot study, which was based on the 
work of Clark and Johnson (1972), the same fonnat as they had used was 
used, viz. the passage was duplicated, with blank spaces of constant 
length, numbered, and numbered blanks were provided on the right hand 
margin of the page. (See Appendix D) 

Close procedure and paragraph performance: A review of research. 

The major problem facing the use of the cloze as a means of replacing 
multiple-choice tests as a measure of paragraph performance has been the 
lack of a frame of reference by which scores on a cloze test might be 
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interpreted. Although a higher score for one individual obviously 
indicates that he has performed better than one "who has obtained a lower 
score, the absolute figures (i.e. the raw cloze percentage scores) do not 
tell us how well the readers comprehend the material. Likewise, it is 
reasonable, on the surface at least, to say that a higher mean score for 
one set of material indicates that it is of an easier standard than 
material that obtains a lower mean score, but this does not tell us much 
about the actual difficulty of the material. 

In order to overcome this problem attempts have been made to determine 
comparable cloze and multiple-choice comprehension test scores, especially 
in relation to 75$ and 90$ levels of comprehension. By doing this it is 
believed that passage performance criteria can be established that will 
allow teachers to use the simpler, mechanical and objective cloze procedure, 
rather than the subjective, problem-ridden multiple-choice process. 

Bormuth (1967 ) 

The earliest work in this area was carried out by Bormuth (1 967). In 
this study a 50 item cloze test and a 31 item multiple-choice test were 
made over nine passages. Each of the multiple-choice tests contained 
questions thought to measure seven different types of comprehension skills. 
Validation was tested by asking two qualified test experts to independently 
classify the items as to type and to discard items, and also by trying out 
the items on 73 children and discarding those items that were negatively 
correlated with the total. 

The passages each contained approximately 275 words and had a 
Dale-Chall readability from 4.5 to 6.5. The exercises were administered 
under untimed conditions to 100 pupils in grades 4 and 5. In each case the 
cloze form of the test was administered first, the multiple-choice form 
being taken three days later. 

Scores for each individual over all nine of the cloze and multiple- 
choice tests were summed to form two sets of scores. A scatter plot of the 
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two sets indicated linearity. The product moment correlation was then 
calculated and the data fed into regression equation to calculate the 
most probable milti pie-choice score associated with each of several cloze 
scores. 

The results indicate that if the conventional passage performance 
criteria are accepted, a passage on which a student receives a cloze score 
of 3^ is sufficiently understandable to him to be used in his instruction - 
i.e. a score of 38$ on a cloze test is equivalent to a score of 75$ on a 
multiple-choice test over the same material. Likewise, a 50$ result on 
the cloze is equivalent to 9C$ on a multiple-choice test. Bornruth also 
provided comparable scores if one demands as a criterion a multiple-choice 
equivalent score corrected for guessing - 43$ and 52$. 

Bonnuth quite correctly warned that the accuracy of his predictions is 
only as good as the cloze test data he had collected, and that it should be 
clearly understood that the comparable scores hold good only where the 
dependent scores are obtained using test instructions and tests similar to 
those used in Ms study - although he doesn't really detail them, parti- 
cularly the instructions. 

Bormuth (1968 ) 

If a follow-up study, Bornruth (1 968) set out to deteimine a set of 
criterion scores comparable to scores on oral reading tests. In this 
study the materials used were paragraphs from the four forms of the Gray 
Oral Beading Tests (1963). Each form contains 13 paragraphs in a graded 
sequence ranging from a very easy pre-primer level of difficulty through 
paragraphs difficult enough to challenge able high school students. For 
the comprehension tests it was necejseuy to augment and revise some of the 
items in the published versions of the tests in order to obtain a reliable 
measure of how well students comprehended each paragraph- The items were 
constructed by using transforations (after Chomsky, 1957) on the language 
in the passages. 
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Two versions of a cloze test were made from each passage by deleting 
different patterns* Subjects were drawn randomly from grades 4-6 in a 
single school. Twro of the four paragraphs at each level were randomly 
assigned to each subject who took these as cloze tests. The complementary 
pair were taken bj each subject as oral reading tests. 

Since oral reading test scores were often available for only a portion 
of the range of paragraph difficulty, ordinary regression techniques could 
not be used to determine the comparable scores. Instead a simple matching 
procedure was used. To find the cloze score comparable to the 75$ 
comprehension criterion, the most difficult paragraph level on which a 
subject obtained a comprehension level of 75$ was found, and the subjects 
cloze score on that level was noted. When no comprehension score of 
exactly 75$ was obtained, the level of paragraph difficulty having the 
score nearest to 75$ was used. The cloze scores were then averaged across 
subjects to obtain the comparable score. 

In fact, the matching procedure used in this study was probably more 
defensible than the regression method in the first study, when a 'goodness- 
of-fit 1 approach would seem to have been more appropriate. 

Cloze scores of 44$ and 57$ were found to be comparable to the 
criterion reference scores of 75$ and 9C$ respectively. These can be 
compared with the 38$ and 50$ of the previous study. The seven point 
difference between the independent level cloze scores in the two studies 
can be explained - according to Bormuth - by the fact that a ceiling effect 
was observed in the multiple-choice scores in the earlier study, and this 
probably suppressed the multiple-choice scores at the upper end of the 
range, thus resulting in an artificially low comparable cloze score. On 
the other hand, the difference might be explained, at least partly, by the 
difference in methods of obtaining equivalence. 

Whilst pointing out that the study needed replication, and that results 
could only be generalized to subjects and passages similar to those in the 
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study, Bormuth believed that aqy replication would obtain s imi lar results 
because most of the items written were written as transformations, thus 
precluding the possibility of them being manipulated arbitrarily to alter 
their difficulties, and because most of the paragraphs were very short 
and the number of items written for every passage was relatively large, 
nearly eveiy item that could have been written for each paragraph was used, 
thus reducing the possibility of bias. 

Rankin and Culhane (1 969 ) 

Rankin and Culhane (l 969) carried out what was essentially a 
replication of Bormuth f s 1967 study. Although there were slight 
differences between the procedures used, the investigation was probably 
comparable in all significant aspects except that Rankin and Culhane used 
only fifth grade children as subjects. 

Although there was fairly close agreement between Rankin and Culhane ! s 
scores and Bormuth's scores at the 75$ and 90$ level, there are considerable 
differences at other levels. (See Table l) 

In fact, taken over the range of 50$ to 100$ multiple-choice scores 
the cloze comparable scores show a range of 39 (l 9—57) in Bormuth 1 s study 
and 65 (10-74) in Rankin and Culhane ! s. 
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Equivalent cloze and multn^le-choice 
percentage scores for Boriu oh (1967, 
1968) and Rankin and Culhane (1969/ 
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50 


19 




10 


+ 9 


55 


23 




15 


+ 8 


60 


27 




22 


+ 5 


65 


31 




28 


+ 3 


i 70 


35 




35 


0 


75 


38 


44 


41 


* 3 


80 


42 




48 


- 6 


85 


46 




54 


- 8 


90 


50 


57 


61 


-11) 


95 


53 




67 


-14 


100 


57 




74 


-17 



* Note that in Bormuth 's 1968 study comparable scores were only given 
for the 75# and 9C$ levels* 



Rankin and Culhane point out that the average difference is in fact 
only 3»1 percentage points, but this is not very convincing, Rankin and 
Culhane argue that the greatest discrepancies lie in the scores comparable 
to multiple choice scores of 85 and above, and that this may be accounted 
for by Bormuth ! s belief that ceiling effects gave him artificially low 
comparable cloze scores in the upper levels. However the difference column 
(Table 1 ) with its increasing differences at the extremes of the range 
exhibits all the manifestations of the typical regression effect, and this 
is a more likely explanation of the differences. 
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There is reasonable correspondence between Bormuth's 1968 scores 
and Eankin and Culhane' s scores at the 75$ and 90$ levels - 44/41 and 57/61. 
On this basis Rankin and Culliane saw fit to say: 

'It is now possible for teachers to interpret cloze test results 
with some degree of confidence by using specific percentage 
scores as criteria of acceptable performance. The use of the 
comparable cloze and multiple-choice scores found in this study 
should be particularly useful for a teacher who wishes to 
measure reading comprehension of pupils in a specific subject 
matter field by using a cloze test based on material in that 
field. 1 (p. 198) 

Anderson and Hunt (1972 ) 

The only other published study in the development of comparable cloze 
and multiple-choice scores is that of Anderson and Hunt (1972). This study 
was carried out with children in schools in Papua Hew Guinea who had learned 
English as a second language. Although Boimuth's basic approach was used 
there were some differences. There is no indication of the length of the 
passages, except that they were 'shorty and whereas Bomuth (1 967) and 
Rankin and Culhane ( 1 969 ) both used 31 item multiple-choice tests per 
passage f Anderson and Hunt used 90 items over the nine passages used - 
i.e. an average of ten items per passage. There is no indication of the 
validation of these multiple-choice items. Also, whereas Boimuth (1967, 
1968) and Eankin and Culhane (1969) had used a deletion rate of every 
fifth word, Anderson and Hunt used an eveiy eighth word deletion rate. 

Anderson and Hunt achieved comparable cloze scores of 44$ (for 75$ m-c) 
and 53$ (for 9C$), and come to the conclusion that the agreement between 
their scores and those carried out in a different countiy and within a 
different educational system seems remarkably close. They conclude by 
claiming that although the criteria they derive and those previously 
derived by Boraruth and Rankin and Culhane will not be applicable in all 
future cloze and multiple-choice comprehension tests, the results should 
enable primaiy school teachers to use their results from cloze tests with 
confidence to judge the suitability of reading materials for particular 
pupils. 
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TABLE 2 




Summary of cloze comparable criteria 




for the four studies 




Multiple-choice 


criteria 


15$ 


90$ 


Bormuth (1967) 38# 


50$ 


Boirauth (1968) 44^ 


57$ 


Rankin and Culhane (1 969) 41$ 


6\$ 


Anderson and Hunt (1972) 44$ 


53$ 



2 summarizes the comparable cloze and multiple-choice criterion 



scores for the four studies discussed. 



Mosberg. Potter and Cornell (1968 ) 



ERIC 



Relevant to the above studies is the investigation carried out by 
Mosberg, Potter and Cornell (1968) into the relationship between cloze and 
multiple-choice tests. Working at two grade levels - grades 5 and 8 - with 
reading passages at difficulty levels either two years below, two years 
above, or at subject's grade level, they tested at each grade level and each 
passage difficulty level a large number of reading passages with a large 
subject sample. 

Table 3 shows the obtained correlation co-efficients between cloze and 
multiple-choice performance. 



TABLE 3 

Correlation between cloze and multiple-choice 
performance according to grade and difficulty level. 

(after Mos berg. Potter and Cornell) 

Difficulty level Grade 5 Grade 8 

??• « 649 -190 

.429 .367 

0vera11 .535 'Ml 

it 
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The correlations reported in Table 3 above suggest that although the 
cloze procedure does measure some component of comprehension as measured 
by multiple-choice tests, there is a large component measured by the 
multiple-choice tests which is not accounted for by the cloze. Mosberg 
et al do point out that their correlations were calculated on the basis of 
matched pairs, and that insofar as these were not perfect the correlations 
are depressed. However they do feel constrained to say that they are 
cautious in their acceptance of the cloze procedure as a predictor of what 
a student would score on a multiple-choice test. This study deserves 
replication. 

Bormuth (1 971 ) 

The four studies reported above are based on acceptance of the 
frustration, instructional and independent levels of reading. In fact 
there appears to be no empirical evidence to support these three levels, 
i.e., that although the traditional criteria of 75$ and 90$ have been 
widely accepted by reading researchers and teachers, there is no evidence 
that they are any more than operationally defined levels. Powell (1968), 
Hunt (1969), and Spache (1969) consider these Killgallon-Betts Criteria to 
be arbitrarily fashioned and not commensurate with reality, although their 
major argument is with word recognition criteria rather than with the 
multiple-choice comprehension criteria discussed in this paper. Spache 
however believes the 75$ for instructional reading level should only be 
about 60$. 

Because he believed these criteria, if not arbitrary, were at least 
unexplicit and unrationalized, Boimuth (1 971 ) set out to establish 
rational passage performance criteria using the cloze procedure. 

Bormuth believed that a reasoned approach to identifying the criterion 
level of performance on a passage would set the score at the performance 
level where a weighted sum of the outcomes showed that a maximum benefit 
was to be expected. In studying the variables affected Bormuth suggested 
that the following were relevant: Cognitive variables such as learning 
and retention and transfer of information in the passage; Proficiency 
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variables such as rate of reading and latency of responses acquired from 
the passage; Affective variables such as students' preferences for the 
subject matter, style, the difficulty of the passage and the students 
willingness to study it; Economic factors such as the costs involved in 
preparing suitable materials; and Psychosocial factors such as the 
effects on self concept of having to study materials at the given level 
of difficulty relative to the subject's level of ability. 

In the series of studies reported in his 1971 paper, Bormuth included 
only the following factors in his criterion selection model: Measures of 
information gain, rate of reading, willingness to study, preferences for 
the subject matter, style and level of difficulty. 

In these studies he set out to establish: 

(a) the regressions between each of these variables and cloze scores; 

(b) a set of weights representing the relative values -laced upon 
each of these variables; " 

(c) what variables influenced the shapes of the regressions and 
therefore required a differentiation of the passage 
performance criterion score. 

Initially the studies were designed to permit the results to be 
generalized to students in grades 3 - 12, to materials on most of the topics 
and at most of the difficulty levels that these students would be likely to 
encounter in instruction, and to each of the major purposes for which 
students are likely to read a passage. However, because cloze and grade 
level consistently interacted in all the regressions, it was necessary to 
identify different criterion scores at each grade level. Also because 
students assigned different ratings to materials depending upon whether 
they were to be used for textbook, reference, or voluntary reading purposes, 
it was necessary to allocate criterion scores for each of these three 
purposes at each grade level. 

As a result, Bormuth comes up with a set of scores for each grade 
level as shown in Table 4. Only grade 3 scores are used here for 
illustrative purposes. 
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TABLE 4 

Cloze scores and dependent behaviour efficiency for 



Criterion 


Cloze 
score 






Dependent Behaviours 








Info 
^ain 


Rate 
£dg_ 


Subject 
matter 


Style 


Difficulty 


Textbook 


54 


81 


59 


100 


99 


55 


Reference 


52 


78 


57 


100 


98 


47 


Voluntary 


62 


90 


68 


97 


99 


1 



The figures in Table 4 are interpreted in this way: 
a cloze score of 54 on a passage from a textbook may be regarded for grade 3 
children as producing an efficiency rate of 81$ on information gain, 59$ 
on rate of reading, 100$ on subject matter, etc. Boimuth does not make 
very clear what he means by efficiency rate, and although an 81$ efficiency 
rate on information ga:Li seems a reasonable statement, 100$ efficiency rate 
for subject matter or 99$ efficiency rate for style, is not readily 
meaningful. 

Boimuth believes that although, the scores he presents are only a crude 
first approximation to those ultimately sought as passage criterion, they 
are probably much superior to any other passage criteria in use. Thus, 
whilst cautioning practitioners and researchers about using them without 
considerable caution, since they contain both systematic and random error, 
he does suggest that they be used. 
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Summary 

Descriptions have been given of two different ways in which the cloze 
procedure has been used to obtain passage performance criteria: 

(a) By establishing comparable cloze scores for multiple-choice 
test performance (Bormuth, 1967, 1968; Rankin and Culhane, 1969; and 
Anderson and Hunt, 1972). The assumption lying behind these studies is 

44 
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that if you accept the traditional 75# and 90$ levels of performance as 
indicating instructional and independent levels of reading, it is better 
to use the established equivalent cloze scores as the measure, because the 
close method of measuring comprehension is simpler, the mechanical deletion 
of words is an objective procedure and does away with all the problems 
associated with subjectivity and the difficulty of items in multiple-choice 
tests. 

(b) By establishing completely new passage criteria and using cloze 
scores as the direct measure. (Bormuth, 1971) 

Research implications. 

Whether one accepts the approach of (a) or of (b) above, in both cases 
the criterion score is established as a single score. For example, if one 
takes Bormuth' s 1968 criterion of 44# as indicating the instructional level, 
this means that if a child scores less than 44# on a. passage thought to be 
representative of that material, then the passage is too difficult for him, 
or if he scores 44^ or above, it is of suitable difficulty. 

There are two problems associated with this approach: 

(a) The assumption is that any one cloze test constructed over a 
given passage of material is equivalent in difficulty to any other cloze 
test constructed over the same passage. If the cloze deletion pattern used 
is an every fifth word deletion, there are five possible cloze tests that 
can be constructed, if every seventh word, ttere are seven possible cloze 
tests, and so on. As there is no necessary consistency in the English 
language as to the length of sentences, the position of words in sentences, 
and the relationship of words to one another within sentences, it does not 
necessarily follow that any one deletion pattern will be of the same 
difficulty as any other deletion pattern within the same paragraph. This 
would not matter if the scores were being used simply to rank the children 
doing the test in some particular order, but when the score is being used to 
relate the performance of the child to a single score criterion, then the 
actual difficulty of that particular cloze test as compared to any other 
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cloze test that could have been constructed over the same material is a 
question that needs to be answered* 

The purpose of the pilot study (Chapter 3) is to investigate this 
matter* 

(b) Secondly, the use of a single score criterion for any material 
and any deletion pattern, suggests a precision that is unreal. The 
purpose of the main investigation (Chapter 4) is to establish operationally 
the range of appropriate scores rather than a single criterion score* 
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CHAPTER III 
PILOT STUDY 

The purpose of the pilot study reported in this chapter was to 
determine if it could be predicted that any one of the possible 
alternative cloze fonns of a passage could be significantly easier, or 
more difficult, than the other forms. 

Factors determining difficulty of word replacement . 

There are a number of ways in which deleted words could be 
categorized in terms of their possible difficulty of replacement* 

Parts of Speech . 

Parts of speech influence comprehension (Huus, 1968), and Bormuth 
(1966) has shown that the ratio of pronouns to conjunctions is a good 
predictor of difficulty, Louthan (1965) found that if prepositions, 
conjunctions or pronoun substantives are deleted, there is no appreciable 
difference between the performances on tests following the cloze materials 
and those following unmutilated passages, whereas specific verb deletions, 
noun and modifier deletions lead to marked loss in comprehension. Elley 
(1969) found, with a sample of secondary school students, that prepositions 
and pronouns were the easiest to replace in the particular cloze exercise 
he used, and that nouns were the most difficult. In fact, as a result of 
his studies, he proposed a noun frequency count as an appropriate means 
of deter mining the 1 difficulty 1 - and hence the readability - of reading 
materials. 

Length of Word . 

Long words have often been thought to be more difficult, and the 
number of syllables is a common element in many of the generally accepted * 
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readability formulas (See p. 6). Coleman (1967) found high correlations 
between difficulty and number of letters, number of syllables, and 
number of affixes, stems and inflexional morphemes. Correlations found 
between number of syllables and passage difficulty include 0.44 (Gray and 
Leary, 1935), 0.69 (Flesch, 1950) and 0.63 (Bormuth, 1966). 

Familiarity of Words. 

If it is assumed that meaningfulness is largely an outcome of 
frequency of exposure then it can be argued that the comprehension 
difficulty of a passage will be strongly affected by the number of 
unfamiliar words included. Some support for this is given by a number of 
studies. Dale and Chall (1948) in their reading difficulty study found 
that of the five indices they used the highest correlation with their 
criterion was the proportion of words outside the Bale list. Spache 
(in Hunnicut and Iverson, 1968) obtained a correlation of 0.68 in a 
similar study. Gray and Leary (1935) found that the factor most closely 
correlated with reading comprehension for poorer readers was the number 
of familiar words in the material. Lorge (1948), Forbes (1952) and 
Bormuth (1966) have all found similar relationships between familiarity 
of words and difficulty of comprehension. 

Elley (1969) suggests that this relationship is further strengthened 
by the fact that the measure of familiarity is relatively weak. The words 
in the passages are classified as either familiar or unfamiliar, with no 
intermediate categories. "Since correlations which depend on a two-unit 
scale are usually lower than those based on a graduated scale, it would 
seem logical to conclude that a more refined measure of familiarity would 
make for an improved predictor of readability." (p. 41 4) 

Word categories used in this study . 

The evidence discussed above suggests that there is justification for 
investigating the ease or difficulty of replacing deleted words. For this 
purpose four word categories, each with sub-divisions, were determined. 
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(a) The number of letters per word. This was sub-divided into four; 
1-2, 3-4, 5-6, and 7 or more letters. 

(b) Number of syllables per word. This category was sub-divided into 
three; 1, 2, 3 or more syllables. 

(c) Whether the word was 'In' or 'Not in« common word lists. For 
this purpose a composite list of 'key', 'basic', 'instant' and 
■sight' words compiled from the lists of Edwards and Gibbon (1964), 
Pry (1968), Kucera and Francis (1967), McNally and Murray (1962) 
and Rinsland (1945) was used. In total this list included 344 
words, including words such as 'and', 'but' and 'came', which 
were in all five lists, and words such as 'woman', 'those' and 
•yet' which were included in only one of the lists. The complete 
composite list is included as Appendix B. 

(d) The part of speech. Bight sub-divisions were used; adjectives, 
adverbs, articles, conjunctions, nouns, prepositions, pronouns 
and verbs. 



Determination of 'easy' categories. 

Clark and Johnson (1972, Appendix B, pp. 23-25) report in detail part 
of the results of their investigation. Included are the percentage errors 
made by a sample of 55 grade 6 children in Victorian metropolitan schools 
in replacing words deleted from a passage from Doug of Australia (Cavanna, 
1965). For this cloze exercise every eighth word, conmencing with the 
first, was deleted. This was the data used in this particular pilot study 
to determine the 'easy' categories of words to replace, and then to predict 
what would probably be the 'easiest' and 'most difficult' of the possible 
alternative cloze forms of the passage. The Clark and Johnson data is shown 
in Appendix C. 

Table 5 shows the percentage of correct responses for each of the sub- 
divisions of each of the four chosen categories using the data of Clark and 
Johnson (1972). The figures indicate quite clearly that for these subjects, 
with this particular passage, the easiest sub-categories of words to replace 
were those words that were 1 or 2 letters long, and/or were of one syllable, 
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and/or were 'In 1 common words lists, and/or were 'structural ! / f functional 1 
words - articles, conjunctions, prepositions or pronouns. 



TABLE 5 

Percentage of correct responses 
in each of the four categories . 



Category Percentage 

1 . Length of words 

1-2 letters 61.9 

3-4 letters 47.1 

5-6 letters 36.5 

7 or more letters 21.1 

2. Number of syllables 

1 syllable 48.8 

2 syllables 26.3 
More than 2 syllables 24.1 

5. Words in common words lists 

In lists 52.2 

Not in lists 27.1 

Am Parts of Speech 

Adjectives 24.8 

Nouns 38.2 

Adverbs 38.8 

Verbs 43.9 

Articles 44.0 

Conjunc tions 51.8 

Prepositions 54 . 5 

Pronouns 67.6 



Predicting Easiest 1 and 'most difficult 1 cloze versions . 

An every seventh word deletion cloze test was then prepared using the 
same passage. Each of the deleted words from the seven possible cloze tests 
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was then placed into the appropriate categoiy. Table 6 shows the number of 
words in each of the subdivisions of each of the categories for each of the 
seven deletion patterns. 



TABLE 6 














Number of words in each category for 
each of the possible cloze versions. 








Category 

Pattern 1 


2 


Number of words 
3 4 5 6 


7 


1 . Length of word 














1-2 letters 8 


10 


8 


15 


8 


10 


12 


3-4 letters 23 


22 


23 


18 


21 


26 


23 


5-6 letters 5 


13 


10 


12 


9 


11 


10 


7 or more letters 1 4 


5 


9 


5 


12 


3 


5 


2. Number of syllables 














1 syllable 34 


39 


33 


39 


35 


41 


40 


2 syllables 7 


9 


13 


8 


8 


8 


8 


More than 2 syllables 9 


2 


4 


3 


7 


1 


2 


3. Words in common word lists 














In lists 30 


30 


29 


37 


32 


37 


36 


Not in lists 20 


20 


31 


13 


18 


13 


14 


4. Fart of si»ech 














Adjectives 5 


6 


5 


6 


7 


5 


5 


Nouns 16 


13 


15 


8 


12 


11 


9 


Adverbs 1 0 


11 


12 


7 


18 


8 


7 


Verbs 3 


2 


5 


6 


1 


6 


6 


Articles 4 


4 


3 


4 


4 


5 


6 


Conjunctions 1 


2 


3 


5 


2 


3 


1 


Prepositions 7 


6 


5 


10 


3 


4 


10 


Pronouns 4 


4 


2 


4 


3 


7 


6 



The information in Table 6 indicates that, if the number of 1-2 letter 
words, the number of 1 syllable words, the number of words in conmon word 
lists, and the number of articles, conjunctions, prepositions and pronouns 
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are taken as the criteria for degree of difficulty of replacement, pattern 
4 should be the easiest version and pattern 5 should be the most difficult 
version. !Eable 7 compares these two patterns according to the number of 
words deleted which belong to all four 'easy 1 categories (i.e. the number 
of words that are 1-2 letters and are 1 syllable and are in common word 
lists and are either articles, conjunctions, prepositions or pronouns), 
those that are in any three of these categories, and so on. 



TABUS 7 




Muniber of words in the 'easy 1 categories for 
pattern 4 (easy version) and pattern 5 (hard 
version) from Dou* of Australia. 


Combination NnmVio-r- ne «at.^ 

Pattern 4 


in 

Pattern 5 


All four 'easy 1 categories 11 


3 


Any three 4 


6 


Any two 21 


18 


Any one 7 


11 


None of the 'easy 1 categories 7 


12 



In order to further test the ability to predict the difficulty of 
deletion patterns within the same passage, a passage was chosen at random 
from Jteserts (ooetz, 1956). The excerpt has a Piy (1968) readability 
rating of Grade Six, and was from a primary science reference, whereas 
Doug of Australia was from a children^ novel. 



Using the same method as described above, it was predicted that 
deletion pattern six (i.e. deleting every seventh word commencing with 
the sixth word in the passage) would be easier for the children to do 
than deletion pattern one. Table 8 summarises the difference between 
the two patterns by showing the number of words in each of the 'easy 1 
categories. 
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TABLE 8 




Number of words in each 'easy 1 categoiy for 
pattern 1 (hard version) and pattern 6 (easy- 
version) from Deserts. 


Category 


Number of words 
Pattern 1 Pattern 6 


1-4 letters 


26 


35 


1 syllable 


36 


39 


In common word lists 


26 


33 


Conjunction/ article 
Preposition/ pronoun 


15 


20 



As a result of identifying what appeared to be the easiest and the 
hardest versions for each of two passages, it was decided to experimentally 
test the following hypotheses. 



Experimental Df sign 
Hypotheses 

1 . That the mean score for grade 6 children doing a cloze test on the 
passage from Doug of Australia with every seventh word deleted 
commencing with the fourth word will be significantly higher than for 
those doing a cloae test for the same passage with every seventh 
word deleted commencing with the fifth word. 

2. That the mean score for grade 6 children doing a cloze test on the 
passage from Deserts with every seventh word deleted commencing 
with the sixth will be significantly higher than those doing a 
cloze test for the same passage with every seventh word deleted 
commencing with the first word. 

Procedure 

Subjects for the experiment were 196 grade six children from eight 
Melbourne metropolitan primary schools. All schools used were in south- 
eastern suburbs. Sex distribution was approximately equal. 03ie Doug of 
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Australia passage was done by 106 children (53 for each pattern) and the 
Deserts passage by 90 children (45 for each passage). The difference in 
numbers is due to only one passage being used in one of the grades, where 
half the grade did the cloze test and the other half did an alternative 
task* 

The cloze tests were prepared by the duplication method, with the 
blanks numbered and numbered blanks provided on the right hand side of the 
page. (See Appendix D for sample) 

The four experimental cloze tests were randomly distributed to the 
children in each grade* When this had been done children were shifted so 
that no child was sitting next to another who was doing the same test, or 
a test from the same passage. This procedure was carried out because the 
author has found that children become aware of the fact that the answers 
to their deletions are in the text of the alternate foim being done by the 
person next to them and there is therefore a tendQncy for some children to 
cheat. Thus, unless this provision is made, spurious and quite misleading 
results can be obtained. 

The administration of the tests for this pilot study was carried out 
by nine third year Diploma of Teaching (Primaxy) students from State College 
of Victoria, Toorak. These students attended a briefing session with the 
author before going to the classrooms to administer the tests, aixl vere also 
given written instructions as to the procedure to follow. (See Appendix E) 

Results 

Table 9 shows the mean replacement scores for each of the two 
experimental patterns for the two passages. As 50 words were deleted for 
each of the cloze tests, the highest possible score for any subject was 
50. 
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TABLE 9 

Mean number of correct replace- 
ments for the two passages. 

Doug of AnafrraHn Deserts 

Pattern 4 



Mean 25.71 

S.D. 8.49 

Variance 72.06 

n 53 



Pattern 5 
(hard) 


Pattern 6 
(easy) 


Pattern 1 
(hard) 


16.39 


18.10 


13.70 


7.11 


10.36 


8.S5 


50.55 


107.33 


80.10 


53 


46 


46 



For the passage from Doug of Australia the difference between the mean 
scores of 25.71 ('easy') and 16.39 ('hard') was significant at the .001 
level with a t of 6.128. The difference between the subjects' performances 
on these twa close versions of the same passage is reflected by the fact 
that for ths 'easy' passage 22 of the subjects obtained scores of 30 and 
above, whilst only two of the subjects doing the 'hard' passage obtained 
similar scores. The actual ranges of scores obtained were 7-47 for the 
'easy' passage, and 1 - 34 for the 'hard' passage. 

Although the passage from Deserts had been rated at Grade 6 level by 
the Pry (1968) readability graph, the children in the sample used for this 
study found it very difficult. For the predicted 'easy' pattern the mean 
replacement was only 18.1 (a replacement rate of only 36.20#), with a 
relatively large standard deviation of 10.36 and a range of scores from 
7-47. For the •hard' pattern the replacement rate was only 27.4#, and if 
the two highest scores for this pattern, which were 13 above the next highest 
score, are taken out, the mean correct replacement rate falls to 24. 6& 

However, despite the overall difficulty of this passage, the hypothesis 
that the 'easy- pattern would yield a significantly higher mean replacement 
score than the 'hard' passage was supported, with at of 2.180 which was 
significant at the .05 level. 



45. 



For both the passages chosen for this pilot study, the mean score 
obtained by the children doing the predicted 'easy 1 pattern was signifi- 
cantly higher than that for the children doing the predicted 'hard 1 
pattern. Thus both of the hypotheses are supported. 

For the detailed investigation of the rate of correct replacement of 
words according to the four categories, only the results of the Doug of 
Australia passage were used* The Deserts passage, was not used because the 
overall difficulty was such as to suggest that insufficient useful 
information would be obtained to warrant the time involved in a detailed 
investigation of the results. 

Table 10 shows the percentage of correct replacements for each of the 
categories for the two experimental cloze tests from the Doug of Auatralia 
passage. These figures clearly indicate support for the predictions made 
regarding the difficulty of replacing deleted words. 



TABLE 10 

Percentage of correct replacements for each categoiy for students 
doing the two experimental patterns from Doug of Australia. 



Category 

1. Length of words 
1-2 letters 
3-4 letters 
5-6 letters 
7 or more letters 

2. Number of syllabi eg 

1 syllable 

2 syllables 
More than 2 syllables 

3. Words in camion word lists 
In lists 
Not in lists 

4. Parts of speech 
Prepositions and pronouns 
Conjunctions and articles 
Adverbs and verbs 

Adjebtives and nouns 



Percentage correct replacements 



Pattern 4 

( ■ 



73.2 
58.6 
40.6 
25.5 

61.2 
31.9 
36.2 

64.5 
27.6 

71 „6 
60.5 
45.3 
40.8 



Pattern 5 
(hard) 

56.5 
36.8 
26.7 
16.6 

39.1 
15.3 
22.4 

43.7 
16.8 

53.4 
61.8 
21.4 
22.7 
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As can be seen from Table 10, in both cases the easiest categories 
of words to replace, judged on the percentage of correct replacements, 
were 1-2 letter words, 1 syllable words, words in cannon word lists, and 
prepositions, pronouns, conjunctions and articles (or functional/structural 
words). 

In both versions there is a slightly higher mean percentage replacement 
rate for 'more than two syllables' than for 'two syllables'. In both cases 
the number of words in the 'more than two syllables' subdivision was small - 
three for pattern 4 and seven for pattern 5. In pattern 4 one of the words 
- 'aborigines', and in pattern 5 two of the words - 'witchetty' and 'another' 
were highly redundant in the context and their high replacement rate pushed 
up the mean. Excluding these three words the mean percentage correct would 
have been 13.7 instead of 36.2 and 22.4. 

Of the fifty words deleted in pattern 4, eleven fell into all four 
'easy' categories. There was a 76.9^ replacement rate for these words, 
whilst there was only a 27. 6# replacement rate for the seven words that 
could not be fitted into any of the four 'easy' categories. For pattern 5 
the same pattern appeared, with a 59.8^ replacement rate for words in all 
four 'easy' categories and 13.6# for those not in any. 

Summary 

Pour ways of categorising words deleted from passages when using the 
cloze procedure were determined. Using these as a basis, the percentage 
errors made by subjects in the study of Clark and Johnson (1972) were 
computed. These figures showed that for those subjects the easiest words 
to replace were words that were:- 

(a) 1-2 letters 

(b) of 1 syllable 

(c) that are in common word lists 

(d) that are functional/structural words. 

All the words for each of the possible seven word deletion patterns 
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for 350 word passages from Doug of Australia and Deserts were categorized. 
The 'easy* word categories were used to predict the 'easiest' and -hardest' 
patterns for each of these passages. Tests using these patterns were then 
given to grade 6 children in eight Melbourne metropolitan schools. Mean 
correct replacement scores for the 'easiest' patterns were significantly 
higher than those for the 'hardest' patterns. The percentage of correct 
replacements for the two versions of the two passages supported the chosen 
categories as being the 'easiest'. 

Conclusion 

The results of this pilot study seem to indicate that it is quite 
possible that the difficulty of cloze tests over the same passage using a 
given n*h TOrd deletion can differ significantly, depending upon what 
particular words are chosen for deletion. For example, if there are 
approximately 350 words in a passage and every seventh word is deleted, 
there are seven possible groups of words that can be deleted. Although 
these are equivalent forms in theory, they are not necessarily equivalent 
in fact - their difficulty levels might be quite significantly different. 

This difference in difficulty levels probably does not matter when 
results of close tests are simply used to rank children. If all the 
subjects have been treated in the same way the particular deletion pattern 
probably makes little difference in the rank order. But when the cloze 
test is for the purpose of obtaining a score which is then to be inter- 
preted, as it is with comparable cloze and multiple-choice scores, or the 
cloze criterion scores of Bormuth (1971) and that interpretation is based 
on a single criterion score, then the level of difficulty of the particular 
cloze 'eletion pattern used is of importance. 

The findings of the pilot study suggest that there is a need to 
determine ways of overcoming this difficulty. Two possible means of 
dealing with the problem are:- 

(a) some simple method by which the class teacher could adjust the 

scores on the test to make allowance for the degree of difficulty 
of the test; 



(b) the determination of a range of scores as the criterion for 
interpreting performance rather than the use of a single 
score* 

The purpose of the main study reported in the next chapter is to 
investigate these two possibilities. 
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CHAPTER IV 
EXPERIMENTAL DESIGN 

As the pilot study had indicated the possibility that any one cloze 
test from a particular passage might be more, or less, difficult than any 
other cloze test from the same passage, it was decided to attempt to 
establish the following: 

1 . The characteristics of omitted words which influence the difficulty 
levels of cloze tests. 

2. A simple means whereby classroom teachers could adjust the obtained 
cloze score for any Individual, the adjustment to be related to the 
relative difficulty or ease of replacement of the words deleted in 
the particular cloze pattern used for the test. 

3. An operationally determined cloze criterion score which would 
indicate whether material was suitable for a child's independent 
or unsupervised reading. This present investigation concentrated 
only on the independent level of reading, i.e. the level associated 
with a 9CJ* minimum performance on a multiple-choice test on the 
material, because of the large number of cloze tests required to 
determine this criterion score effectively for any level. 

In order to achieve these the following procedures were used:- 

1 . Using the means of categorizing deleted words developed in the 
pilot study, the replacement rates for all possible deletion 
patterns from a large number of passages were determined. 
In this way 'easy 1 to replace and 'difficult 1 to replace 
categories of words could be determined, and thus the character- 
istics of omitted words which influence the difficulty level of 
cloze tests could be established. 
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2. Using the 'easy 1 to replace categories determined in (l) above 
as predictor variables, and the percentage replacement scores as 
the criterion variable, the number of words in each of the 'easy 1 
categories for each test, together with the percentage replacement 
score for each test, were entered into a multiple regression 
analysis. By this means a formula, or formulas, could be 
established which would allow the teacher to adjust the obtained 
cloze score for any individual in terns of the numbers of words 
with certain characteristics in that particular test. 

3» By determining the mean replacement score for a large number of 
cloze tests, involving all the possible deletion patterns of a 
number of different passages estimated to be at the independent 
level of reading for the subjects, an operationally determined 
cloze criterion score could be established which would indicate 
whether material was suitable for a child's independent or 
unsupervised fading. The standard deviation associated with this 
mean score would give a range of scores which, used in conjunction 
with the mean criterion score, would indicate the efficiency, or 
relative inefficiency, of a single criterion score. 

To meet the needs of the procedures outlined above, each of the 
possible every seventh word deletion patterns from 16 different 350 word 
fepprox.) passages from books estimated to be at the independent level of 
reading for the children involved were used. Thus 112 different cloze tests 
were devised, with 5,600 words deleted. 

The Instruments 

!Bie cloze tests were devised in the following manner, 
(a) Four grade six teachers in Melbourne metropolitan schools were chosen 
on the recommendations of lecturers from the State College of Victoria at 
Toorak and school principals. The grounds for the reconmendations were 
that these four were excellent teachers, had taught for at least ten years, 
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and had an interest in the teaching of reading. Two of the four were the 
reading co-ordinators for the upper grades in their respective schools, 
whilst the other two were responsible for co-ordination of all subjects in 
grades five and six at their schools. 

(b) Die four teachers were asked to rank the children in their grades on 
the basis of their reading comprehension ability. Two of the teachers did 
this on the basis of their knowledge of the children's ability, and as this 
ranking was done in early December it would be expected that the teachers 
would know the children well and that the ranking would be reasonably 
reliable. The other two made their rankings on the basis of their knowledge 
of the children's performance together with information from a series of 
reading tests given during the year. 

(c) The first 28 children in each grade were then divided into four groups 
of seven, the first seven in order being designated Group: tf the second seven 
being designated Group 2, and so on. The members of these groups were 
judged to be of relatively equal ability although obviously there was acme 
spread, with that spread most likely to be most pronounced in Group 1 (the 
best readers) and Group 4 (the poorest readers) for each grade. It should 
be pointed out however that all grades were in excess of 28 and that the 
poorest readers were not included in the investigation. A cross check with 
the comprehension test scores for the two grades for which these were 
available indicated that the ranges, for those two grades at least, were 
not excessive in any group. 

(d) The teachers were then asked to choose one book for each group that 
they considered would be at the independent level of reading for the 
children in that group, i.e. they were asked to choose books that the 
children could be expected to read and comprehend without assistance. 

(e) In all cases the book chosen was an anthology of selections. Thus, in 
each case the passage chosen for the cloze tests was taken from a stoiy 
chosen at random from those in the book. A list of the books chosen, and 
the passages used, can be found in Appendix P. In all cases except one, 
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where the story was little more than 350 words long, the passage chosen 
allowed a 'run-in 1 of somewhere between one sentence and one paragraph 
before deletions commenced, thus allowing some experience of the flavour 
and tone of the passage. 

(f ) Seven deletion patterns were then prepared for each passage by- 
deleting words 1, 8, 15, etc., words 2, 9, 16, etc. etc. In determining 
the 350 words for each passage the following decisions were made. Words 
such as 'I'll 1 , "it's 1 and f we f re f were counted as one word: where words 
were hyphenated, such as 1 co-worker 8 , 'fore-flippers 1 , 'whip-poor-wills 1 
and 'tree-tops 1 , each of the units was treated as a single word; where 
numbers appeared in the text, e.g. "in the winter of 1774" and 

"a 64-gun salute", these were each counted as one unit. Thus 1774 and 64 
were each counted as single words. 

(g) Each of the words was then allocated to its appropriate subdivision 
of each of the four categories. The categories of number of letters, 
number of syllables and »in» common words lists were handled in the same 
way as in the pilot study. For the part of speech category the part of 
speech of each of the 5,600 words was determined ty the investigator with 
The Concise Oxford Dictionary as the basic source of reference. Random 
checks were made of the categorizations by two senior lecturers in English. 
Instead of using all eight subdivisions previously used (see p. 38 ) 9 only 
two subdivisions were used, viz. parts of speech found to be 'easy 1 to 
replace in the pilot study, and an 'other' group. The 'easy' subdivision 
was comprised of personal, personal possessive and relative pronouns, 
prepositions, conjunctions and the definite articles 'the', 'a' and 'an'. 
All other words were placed in the 'other' or 'hard' to replace subdivision. 

The part of speech category was by far the most difficult to use in 
that many words can be more than one part of speech, depending on the 
particular usage, and the line between two possible parts of speech is 
rather fine in some cases. Thus the possibility of placing a word in the 
wrong subdivision is much greater in this category than it is in any of 
the others. 
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Appendix 3 gives an example of the categorizations* It shows the 
categorization of the seven patterns from the passage from The Musical 
Seal, 

(h) The tests were prepared by photostating the passages, painting out 
the words to be deleted with liquid retype, and placing a number, in series, 
in each of the blanks. A separate answer sheet was provided. Samples of 
the tests and the answer sheet can be found in Appendix H. 

Subjects 

The 112 subjects used in this investigation were grade six children 
from four Melbourne metropolitan State Primary Schools. The total group 
was almost exactly divided between beys and girls, although there were 
variations in this relationship from grade to grade and from stab-group to 
sub-group. 

Test Administration, 

All testing was carried out by the author under noimal classroom 
conditions. 

The seven different patterns for each passage wer •> randomly allocated 
to the members of each group. After the material had been handed out 
changes were made to the seating in the room to ensure that no child was 
sitting next to another doing a test from the same material. 

The following instructions were then read, the children following from 
individual copies;- 

n 0n this page is a reading puzzle. Every seventh word has been 
left out of a paragraph from a book, and a number has been put 
Where each word was left out. Tour job will be to try to solve 
the puzzle t*y trying to guess the words left out. You have 
been given a separate answer sheet to write your answers on. 
The first answer has been written in to show you what to do. 

It will help you in doing this exercise, and the longer one 
we are also going to do, if you remember these things - 

1 . Write only one word for each numbered space. 
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2 # Try to fill eveiy blank. Don't be afraid to guess. 

3. If you find any very hard, leave that one and come back 
to it later. 

4. Your spelling doesn't natter as long as we can tell 
what word you meant* 



The subjects then did the following short practice exercise 

When something hot and something cold 1 brought together, 
he&t will always move 2 the hotter thing to the cooler 

3 . Drop some ice cubes in a 4 of warm lemonade. 
The heat from 5 warm lemonade will go into the 6 
cubes* The lemonade will be cooler 7 some of the heat 
has gone 8 of it. The ice will melt 9 heat has gone I 



Pour minutes were given to complete this practice exercise. The 
correct answers were then given, followed by a brief discussion of the 
reasons for certain words being the correct replacement, in opportunity was 
then given to ask questions. Then the final instructions were given:- 

"We are now going to do a much longer puzzle - there are fifty 
words missing this time. Byeiyone is doing a different puzzle. 
You will see that the missing words have been replaced by 
numbers and that the space will gLve you some clue as to the 
length of the missing ward. 

We are trying to find out how boys and girls like you, in a 
number of different grades in a number of different schools, 
can do puzzles like these. Please try your hardest . n 

No time limit was set for completion of the tests. The children handed 
in their sheets as they were completed or satisfied they had replaced as 
many words as they could. After thirty minutes all remaining tests were 
collected. In all these latter cases the children had replaced as many 
words as they could. 
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number (e.g. 34) or a date Te.g. 1973) missing, rather 
than a word. 



into it # 
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Processing the data. 

Although 112 tests were prepared and allocated to subjects, the results 
of only 110 tests are reported. Two subjects in School 4 Group 2 were 
absent on the day of testing, and as this took place veiy near to the end 
of the school year, the school program did not allow for the testing of < 
these children at a later date. 

All tests were scored on the basis of one point being given for each 
correct replacement of a deleted word. Thus, the highest possible score 
was 50 for any subject. For some purposes the scores have been expressed 
as percentages. Where this is the case it is clearly indicated in the text. 

1 • The score for each of the 112 tests was obtained. Fran these scores 
the mean and standard deviation for each passage and for all the 
passages combined was computed. (See Appendix i) 

2. The number of words correctly replaced for each subdivision of each of 
the four categories was determined for each test # These were then 
slimmed to determine the percentage replacement rate for each of the 
subdivisions of each of the four categories for each passage and all 
passages combined. (See Appendix j) 

3. This data was used to determine the easiest subdivisions of each of the 
four categories. 

4. The number of words for each test in each of the easy subdivisions for 
the four categories - words 1-4 letters long, words of 1 syllable, 
words of 1-2 syllables, words in common word lists P and words that 
were either articles, conjunctions, prepositions or pronouns - 
together with the percentage of correct replacements for each test, 
was entered into a multiple regression analysis. The computer program, 
Program Eegran (Veldman, 1967) was used for this analysis. 



9 

ERLC 



56 



CHAPTER V 
ANALYSIS OF USE DATA 

Characteristics of nm-i tted words and difficulty of replacement . 

All 5,600 words in the 16 passages were categorized according to - 

(a) length of word, 

(b) number of syllables, 

(c) common word lists, and 

(d) part of speech, using the subdivisions devised 
for the pilot study. Mean replacement rates were 
then computed for each subdivision for each 
passage and for all passages. 

(a) Length of uord. 

Table 1 1 shows the number of words in each subdivision of the length 
of word category, together with the number of these words correctly replaced 
and the replacement rate for each passage and for all passages combined. 

TABLE 11 

Replacement of words according to the number of letters. 



Passage 


1/2 


Number of letters ner 


word 


School 1 


3/4 


5/6 


7 






1 . Musical Seal 


75 56 


130 78 


78 27 


67 13 




74.67* 


60.00* 


34.62* 


19.4C* 


2. Paul Severe 


68 49 


131 66 


62 19 


83 21 




72.05* 


50.58* 


30.64* 


25.30* 


3. Loaded Dog 


46 26 


162 89 


81 19 


61 19 




56.52* 


54.94* 


23.46* 


31.15* 


4. Aunt Letty 


47 25 


187 93 


72 22 


44 2 




53.19* 


52.40* 


^0.55* 


4.54* 
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Passage 


1/2 


Number of letters uer word 


School 2 


3/4 


5/6 


7 


1 Thft Plflimnnf 


err 

bb 57 


140 93 


70 33 


74 20 




86.3656 


66.43* 


47.14* 


27.03* 


2. The Smiths 


71 


136 61 


77 21 


65 7 




70.4296 


A A Ol"l/ 

44.85^ 


27 .,27* 


10.76# 


3. Insight 


71 56 




79 21 


68 11 




78.87* 


53.79?6 


26.5836 


16 173^ 

IO ( 1 I/O 


4. Frog Prince 


78 60 


1 Q3 1 


Cr\ op 


19 9 




76.92* 


68 3 erf 


AC cat 




School 3 










1 . Af frhaniatfln 


7C £p 
f 3 DO 


132 79 


86 34 


55 12 




90.66J6 


59.84* 


39.53* 


21.81* 


2. Puddin 1 Thieves 


63 4.4. 


174 71 


53 9 


60 14 




69.8496 




16.98>» 


23.33* 


3# Paddington Bear 


75 61 


14.Q OR 


oy yj 


57 18 




81.33* 


65.77* 


43.47* 


31 .5796 


4. Jack 


69 39 


176 97 


62 18 


4.3 6 




56. 52* 


55.11* 


29.03* 


13.9596 


School 4 










1. Wouldn't Box 


86 7^ 


150 102 


55 22 


59 32 






63. 0C* 


40.00* 


52.24* 


2. Christmas r Pre#*n 




114 57 


42 6 


46 9 




68 1396 


50.0056 


14.2956 


1 Q 


3. Seal Family 


51 30 


176 89 


79 34 


44 7 




58.82* 


50.57* 


43.04* 


15.91* 


4. Rip Van Winkle 


72 38 


169 52 


71 16 


38 1 




52.77* 


30.76* 


14.08* 


2.63* 


TOTALS 


1061 765 


2451 1333 


1096 359 


883 201 




72.10# 


54.38* 


32.75* 


22.76* 



In all cases the 1-2 letter words were the easiest to replace, with the 
next easiest being the 3-4 letter words. In five cases (School 1 passage 3, 
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School 2 passage 4, School 3 passage 4 and School 4 passages 1 and 2 ) a 
higher percentage o£ ? or more letter words was replaced than for 5-6 
letter words, aost of these cases this was probably due to the fact 
that a number of words were repeated a number of times, and although these 
words were relatively long, e.g. princess, Stanley, Matthews, and football, 
they were highly replaceable in the particular contexts. 

The total replacement percentages for 1-2 letter words (72.10), 3-4 
letter words (54.38), 5-6 letter words (32.75) and 7 or more letter words 
(22.76), supports the findings of the pilot study regarding the relative 
ease of replacement according to the number of letters in the deleted word. 

For the purposes of the multiple regression analysis referred to later 
in the chapter, the 1-2 letter and 3-4 letter subdivisions were combined to 
give the -easy' subdivision within the category length of word. 

(b) Number of syllables 

Table 12 shows the number of words in each subdivision of the number 
of syllables per word category, together with the number of these words 
correctly replaced, and the replacement rate for each passage and for all 
passages combined. 



TABLE 12 

Replacement of words a^n^r^ tg the number of gy llablea 

Number of syllables per word. 



School 1 1 



3 plus 



1. Musical Seal 236 14s 77 24 3? g 

62.71* 31.17* 5.41* 

2. Paul Revere 227 122 88 27 35 8 

53.74* 30.68* 22.86* 

3. Loaded Dog 247 128 83 22 20 3 

51.82* 26.51* 15.00* 

4. Aunt Letty 276 131 66 16 8 0 

47.4$* 26.67* 0.0O* 
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Passage 




Number of syllabi 


es per word. 


School 2 


1 


2 


3 plus 


I • lilc OXcLLIDallTt 


222 1 56 


94 41 


34 6 




70. 275* 


43.62* 


17.65* 


fc- a dill I Ulio 


251 1 25 


70 11 


28 3 




49.80^ 


15.71* 


10.71* 


3. Insight 


237 139 


88 18 


25 2 




58.65* 


20.45* 


8.00* 


4. Prog Prince 


291 195 


57 33 


2 1 




67.01* 


57.89* 


50.00* 


School 3 








■ • A - L |J if lip TfHJl 


246 1 65 


71 23 


31 5 






32.39* 


16.13* 


c« ruuvjuui AJ1X© V©S 


249 1 1 9 


74 17 


27 2 




yt*7 nct£. 
4 f •795* 


22.97* 


7.41* 


3. Paddington Bear 


252 171 


69 26 


29 10 




67 •Sfijtf 


37.68* 


34.48* 


4. Jack 


269 141 


70 18 


11 1 




52.42* 


25.71* 


9.09* 


School 4 








1 - Ifortllrin * + H/vr 

■ • MWlLlUii V DUX 


253 183 


72 35 


25 11 






48.61* 


44.00* 


t • ouzels unas rrees 


175 95 


43 6 


28 4 




54.29j» 


13.95* 


14.29* 


3. Seal Eamily 


264 134 


68 17 


18 "5 




50.76* 


25.00* 


16.67* 


4. Hip Van Winkle 


259 87 


75 19 


16 1 




33.59* 


25.33* 


6.25* 


TOTALS 


3954 2239 


1165 353 


374 62 




56.63* 


30.30* 


16.58* 
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In all cases words of one syllable were the easiest to replace, and in 
only one case (School 4, passage 2) was there a higher percentage replacement 
for three syllable words than for two syllable words, and even then the 
difference was negligible (14.29* for three or more syllables, 13.95# for 
2 syllables). 

In many cases the percentage replacement rate for three or more syllable 
words was very low, e.g. 0.00# (School t, passage 4), 5.41# (School 1, 
passage 1), 6.25* (School 4, passage 4) and 7.41# (School 3, passage 2). 
For the passage from the Frog Prince the percentage replacement rate was 
50#, but only two of the words were three or more syllables long. 

The total replacement percentages for one syllable words (56.63), two 
syllable words (30. 30 ) and three or more syllable words (16.58), support 
the findings of the pilot study regarding the ease of replacing words 
according to the number of syllables. For the purposes of the multiple 
regression analysis the replacement scores for both one syllable words, 
and one and two syllable words combined* were used as 'easy' subdivisions of 
the number of syllables per word category. 

(c) Words in common word lists. 

Table 13 shows the number of words in each of the two subdivisions for 
this category, together with the number of words correctly replaced, and 
the replacement rate for each passage and for all passages combined. 
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TABLE 13 






Replacement of words according to whether they were 
'in' or 'not in 1 nonmon word lists. 


Passages 


In common 200 
word lists. 


Not 


in common 200 
word lists. 


School 1 








1 . Musical Seal 


203 137 
67.48^ 


147 


37 

25.18^ 


2. Paul Revere 


187 112 
59.89^ 


163 


45 

27.61# 


3. Loaded Dog 


211 115 
54.51* 


139 


38 

27.34# 


4. Aunt Letty 


226 123 
54.43# 


124 


24 

19.36# 



y 1 



61. 



Passages 


In 


common 200 


Not 


in common 200 


word liata 




School 2 












1. The Claimant 


206 152 


142 




51 






73.08* 




35.92* 




2. The Smiths 


204 115 


146 




24 






56.38* 




16.44* 




^ • xnsignt 


217 137 


133 




22 






oj , 1 4]p 




16.55^ 




4» Frocr Prince 


971 




79 




47 






67.16* 




59.50^ 




School 3 












1 . Afghanistan 


214 


151 


136 




42 






70.56^ 




30.8$ 




2. Puddin ' Thieves 


22.5 


111 


127 




27 










21. 26# 




c.J\ 


169 


1 1 Q 




JO 






73.13* 








4* Jack 


235 


135 


1 1K 

112 




OCT 

25 


— — — 




2 r • *r Or* 








School 4 












1. Wouldn't Box 


236 


173 


114 




56 






73.31* 




49.13* 




2* Christmas Trees 


163 


92 


87 










56.457* 




14.95* 




3. Seal Family 


220 


122 


130 




38 






55.46J* 




29.23* 




4. Hip Van Winkle 


209 


80 


141 




27 






38.28* 




19.15* 




TOTALS 


3458 


2106 


2042 




554 




60.91* 




27.13* 
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The percentage of •in 1 words correctly replaced ranges from a low of 
38.280 (School 4, passage 4) to a high of 73.160 (School 3, passage 3), 
whilst for 'not in 1 words the rates ranged from a low of 14.950 (School 4, 
passage 2) to a high erf 59.5C0 (School 2, passage 4). 

In all cases the percentage of 'in 1 words correctly replaced was higher 
than for f not in 1 words. The overall difference of 32.780 in the rates of 
replacement supports the findings of the pilot study that words 'in 1 common 
word lists are much easier to replace. 

For the purposes of the multiple regression analysis the replacement 
scores for 'in 1 common word lists were used. 

(d) Parts of speech. 

Table 14 shows the number of words in each of the two subdivisions for 
this category, together with the number of words correctly replaced, and 
the replacement rates for each passage and all passages combined. Whereas 
eitfit separate subdivisions had been used in the pilot study, a separate 
subdivision for each part of speech, in this case the words were divided 
into only two subdivisions, articles, conjunctions, prepositions and 
pronouns in one and all other parts of speech in an "other" subdivision. 

The percentage of words in the articles etc. subdivision correctly 
replaced ranged from a hi#i of 81 .620 (School 2, passage 1 ) to a low of 
47.870 (School 4, passage 4), whilst for the 'other 1 subdivision the range 
was from a high of 56.530 (School 2, passage 4) to a low of 21.890 (School 4, 
passage 4). 

In all cases the percentage of articles etc. replaced was higher than 
for •other 1 parts of speech. The overall difference of 29.930 in the rates 
of replacement supports the findings of the pilot study that words that are 
articles, conjunctions, prepositions or pronouns are easier to replace on 
average than are words that are any of the other parts of speech. 

Tor the purpose of the regression analysis the articles etc. subdivision 
was used as the predictor variable. 
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TABLE 14 

Replacement of words according to part of speech 




Passage 


Pronoun/Prep osi Hon 
Con.i\mcti en/Article 




Other 


School 1 












1. Musical Seal 


138 


97 


212 




77 




70.299^ 






36.32* 




2. Paul Revere 


1 .2 .2 


PA 
OO 


217 




69 




66.17# 










3. Loaded Dog 


138 


76 


212 




77 




55.08^ 










A a _ 1 m mm 

4. Aunt letty 


138 


81 


212 




DO 




58.70* 










School 2 












1. The Claimant 




4 4 4 
111 


214 




92 




81.62* 






42.95* 




2. The Staiths 


136 


83 


214 




56 




61.03* 






26.17* 




3 Insiriit 


128 


88 






71 




68.75* 






31.99* 




4» Frog Prince 


166 


125 


184 




104 




75.31* 






56.53* 




School 3 












1. Afghanistan 


115 


84 


235 




109 




73.05* 






46.39* 




<L • rxiaain 1 Tin eve s 


138 


80 


212 




58 




57.96* 






27.36* 




3. Padding ton Bear 


144 


112 


206 








77.78* 






46.12* 




4. Jack 


133 


82 


217 




78 




61.66* 






35.95* 
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Passage 


Pronoun/Preposition 
Con.iunction/Article 




Other 


School 4 








1. Wouldn't Box 


130 106 


220 


123 




81 .54* 




55.91* 


2. Christmas Trees 


103 68 


147 


37 




66.02* 




25.17* 


3. Seal Family 


123 76 


227 


84 




61.79* 




37.01* 


4. Hip Van Winkle 


117 56 


233 


51 




47.87* 




21 .89* 


TOTALS 


2116 1413 
66 • 78^ 


3384 


1247 
36.85* 



Regression analysis 

The individual data was then processed by means of a multiple 
regression analysis using the computer program Kegran (Veldman, 1967). 

The analytic procedure incorporated in the program involves the use 
of multiple predictors and a single criterion. A set of "beta" weights 
are then determined for these predictor variables that will produce 
composite predicted scores which will correlate maximally with the criterion 
variable. 

For the purposes of this present analysis the following 'easy 1 
subdivisions of the four categories were used as predictor variables; 
words of 1-4 letters (Predictor 1), words of one syllable (Predictor 2), 
words of 1-2 syllables (Predictor 3), words 'in' cgbqq word lists 
(Predictor 4) and words that were either articles, conjunctions, 
prepositions or pronouns (Predictor 5). 

The input data is shown in Appendix I. For each test the number of 
words in each of these easy subdivisions is shown under predictor variables, 
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and the actual score obtained by the subject doing that test is shown under 
the criterion. 



Tkble 15 shows the correlation matrix for the variables used in this 
study. 









TABLE 15 












Correlation Matrix 












Predictor Variables 






flTT "HptH nn 




1-4 

letters 


1 SVllfl- 

ble 


ables 


In com- 
mon words 


Prep/Pro 
Conj/Art 


#age 
obtained 


1-4 

letters 




0.7602* 


0.4278* 


0.7667* 


0.5129* 


0.1290 


1 

syllable 


0.7602 




0.5525* 


0.6163* 


0.3018* 


0.0477 


1-2 

syllables 


0.4278 


0.5525 




U.3692* 


0.0127 


0.0125 


In cannon 
words. 


0.7667 


0.6163 


0.3692 




0.5117* 


0.3468* 


Prep/Pro 
Conj/Art 


0.5129 


0.3018 


0.0127 


0.5117 




0.2442** 


*age 

obtained. 0.1290 
Significance: 


0.0477 


0.0125 


0.3468 


0.2442 




* P .01 














** p .05 















Amongst the predictor variables the highest correlation (0.7667) was between 
words 'in' cannon word lists correct and words 1-4 letters long, with the 
correlation between one syllable words and words that are 1-4 letters long 
being only fractionally lower (0.7602). Apart from the correlation 
between articles, conjunctions etc. and words of one or two syllables, 
which was only 0.0127, all the correlations were significant. 

For the correlations between the predictor variables and the criterion 
variable, the highest correlation was that of 0.3486 for the number of words 
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•in' common word lists correct, with the number of articles, conjunctions, 
etc. correct next best (r = 0.2442). The two predictor variables involving 
the number of syllables showed very low correlations with the criterion 
(0.0477 and 0.0125). 



Table 16 shows the cumulative variance for the predictor variables in 
combination, commencing with the best single predictor. The order of adding 
in of predictor variables was determined by the computer. 



TABLE 16 




Cumulative Variance for the best combinations 


of nredictor variables 


Predictor Variables 


Cumulative 


4 (common words) 


0.1203 


1 (1-4 letters) 


0.1658 


5 (Preps etc.) 


0.1786 


2 (1 syllable words) 


0.1888 


1 


0.1892 


3 


0.1894 


1 


0.1895 


2 


0.1896 


1 


0.1896 


2 


0.1897 


1 


0.1897 


2 


0.1897 



An examination of the iteration sequence shows that 12.03* of the 
variance in the total percentage scores is accounted for by the number of 
words «in« common words lists (Predictor 4), and that this is the best 
single predictor. The addition of number of wards of 1-4 letters correct 
(Predictor 1) adds another 4.55*, whilst the addition of the number of 
articles, conjunctions, etc. correct (Predictor 5) adds a further 1.28*. 
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As a result, five possible predictor models were determined, viz: 

Model 1 Predictor 4 
Model 2 Predictors 1 and 4 
Model 3 Predictors 1,4 and 5 
Model 4 Predictors 1 , 2, 4 and 5 
Model 5 All predictors, 

with Model 1 being the best single predictor, 
and Model 2 the best pair of predictors. 



Table 17 shows the correlation, variance, Beta and B weights, and the 
regression constant for each of these five models. 



Correlation, 


Variance, 


TABLE 17 

Beta, B and Regression Constant scores for 
five models. 


Model 


Predic- 
tors 


r 


r2 


Beta 


B 


Beg. 
Const. 


1 


4 


0.3468 


0.1203 


0.3468 


1.2890 


7.6240 


2 


1 

4 


0.4072 


0.1658 


-0.3321 
0.6015 


-1.2066 
2.2354 


16.3955 


3 


1 

4 

5 


0.4266 


0.1820 


-0.3766 
0.5579 
0.1518 


-1.3681 
2.0734 
0.5953 


15.1948 


4 


1 
2 
4 
5 


0.4354 


0.1896 


-0.2887 
-0.1268 
0.5756 
0.1360 


-1.0487 
-0.5283 
2.1393 
0.5332 


23.1296 


5 


1 

2 
3 
4 
5 


0.4355 


0.1897 


-0.2812 
-0.1254 
-0.0157 

0.5786 
0.1288 


-1.0214 
-0.5225 
-0.0998 
2.1504 
0.5051 


26.8868 
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The Beta weights, which are standard partial regression weights, 
indicate the extent to which each variable is utilized in the regression 
equation, whilst the B-wei^rt vector, with the regression constant added, 
gives the information scaled in terms of the raw scores of the predictor 
variables* (Veldman, 1 967 ) ♦ 

The predicted percentage scores for each subject for each model were 
then computed, together with the adjusted percentage scores. An example 
is given. 

Example from Model 2 (Predictors 1 and 4 ) 

Subject Pred 1 Pred 2 Pred 3 Pred 4 Pred 5 Obtained 

percent 

0101 28.000 30,000 45.000 26.000 21.000 52.000 

Predicted percentage score - B 1 X 1 + B4X4 + Regression Constant 

a (-0.3321 x 28.000 + 0.6015 x 26.000) 

+ 16.3955 
= 40.732 

Adjusted percentage score - Obtained criterion score plus the difference 

between the predicted percentage score and 
the mean criterion percentage score. 

m 52.000 + (-40.732 + 48.1455) 

■ 59.4135 

A series of P tests was then carried out. Two of these were concerned 
with the significance of the prediction obtained by using (a) all 
predictors, and (b) the best single predictor (Predictor 4). Both were 
significant at the .001 level. 

The remaining P tests were carried out in order to examine the 
predictive efficiency gained by adding predictors to the equation. Only 
one of these, adding Predictor 1 to the best single predictor (Predictor 4), 
led to significant improvement. 
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Table 18 summarizes the results of the F tests. 



TABLE 18 
F Test Results 



Predictors 


D.F. 


F ratio 


P 


All 


5/104 


4.870 


0.0007 


4 


1/1 06 


14.769 


0.0004 


4+1 vs 4 


1/107 


5.833 


0.0165 


4 + 1+5 vs 4 + 1 


1/1 06 


2.102 


0.1462 


4 + 1+ 5+ 2 vs 4 + 1+ 5 


1/105 


0.982 


0.6752 


4 + 1+ 5+ 2+ 3 vs 4 + 1+ 5+ 2 


1/104 


0.016 


0.8958 



This clearly indicates that Predictor 4 alone is almost as profitab?.e 
as using all five predictors although a significant increase is obtained by 
adding Predictor 1 . However no profit is achieved by adding any of the 
other predictors. 

As the purpose of this section of the investigation was to choose a 
simple means of adjusting the scores to make allowance for the characteristics 
of the words deleted in the particular oloze pattern, the results seem to 
indicate two possibilities. The first is to use the best single predictor 
(Predictor 4 - the number of words in common word lists), whilst the second 
is to use the most efficient pair of predictors (Predictor 4 plus Predictor 
1 - the number of words 1-4 letters long). 

For teachers to use these predictors to adjust obtained cloze scores 
the following procedures would be required. 

1 . Using Predictor 4 alone. 

(a) Determine the number of deleted words in the passage that appear 
in the composite common words list. 
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(b) Compute the predicted score by multiplying the number of deleted 
T*ords in the common words list by 1.2890 and add 7.6240. 

(c) The adjusted score would then be the actual total replacement 
score obtained by the child plus the difference between the 
predicted percentage score and the mean criterion score. 

2. Using Predictors 4 and 1 

(a) Determine the number of deleted words in the passage that appear 
in the composite common words list and the number of deleted words 
that are 1-4 letters in length. 

(b) Compute the predicted percentage score as follows: 

Predicted score = (-1.2066 times the number of words 1-4 letters 
long plus 2.2354 times the number of words in common word lists) 
plus 16.3955* 

(c) Compute the adjusted percentage score by the actual total 
replacement score obtained by the child plus the difference 
between the predicted percentage score and the mean criterion 
score. 

Using Predictor 4 alone would be reasonably simple and would not 
require very much work on the part of the classroom teacher. Using 
Predictors 4 and 1 would only be relatively more time consuming and difficult 
to use. The decision as to whether to recommend their use however is 
dependent on a number of points. 

(a) Predictor 4 accounts for only 12.03# of the total variance, and 
Predictors 4 and 1 together account for only 16.58$. Thus, although 
the formulas that arise out of the weights found for the predictors 
in this investigation are relatively simple, the predictors 
probably account for too little variance to warrant recommending 
the use of these foimula to adjust the obtained scores. 

(b) The argument often voiced against readability foimrulas that they 
are too 'mathematical 1 for teachers to be bothered to use, could 
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easily apply in this case. The mathematics required in the 
formulas arising out of this pare sent study is similar to that 
required in the readability formulas mentioned in Chapter 1. 

(c) It must be remembered that in the case of the strongest 1 
predictor (Predictor 4) the data has been based on the number of 
words in a composite common words list. The use of the adjusting 
procedure arising out of this present study would require the 
teacher to have this particular list on hand. Of the four word 
categories and the five predictor variables used in this study 
this is the only one that requires the teacher to have any special 
information. It is therefore less likely that teachers would make 
use of this information than had the best predictor (s) been the 
number of 1-4 letter words and/or the number of one syllable 
words, both of which are very simple to determine and neither of 
which require further reference to any other information. 

(d) The effectiveness of the use of the weights can be shown tjy 
comparing the standard deviations for the obtained and adjusted 
percentage scores in this present study. 

Table 19 shows the means and standard deviations for the obtained scores 
and the scores when adjusted by the use of the weights for Predictor 4 and 
for Predictors 1 and 4* 



TABLE 19 

Means and standard deviations for obtained and adjusted scores 




Obtained 


Adjusted 




percentage 


percentage 


Predictor 4 Mean 


43.145 


48.145 


S.D. 


14.776 


13.859 


Predictors 4 and 1 






Mean 


48.145 


48.145 


S.D. 


14.776 


13.496 
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An analysis of the results shown in Table 1 9 indicates thf\t for each 
of these the difference between the standard deviation for the obtained 
percentage score and the adjusted percentage score, although giving some 
advantage, is so small as to hardly justify the work involved in using the 
weights to adjust the scores. 

If the best single predictor is used the difference in standard 
deviations is only 0.917, whilst for the best pair of predictors the 
difference is 1.280. There appears therefore that there is little 
justification in expecting teachers to go to all the work involved in 
making the adjustments when overall there is only a small difference in the 
standard deviation. 

Thus it appears as if this investigation has not succeeded in devising 
a simple practical means for adjusting obtained cloze scares in terms of 
the characteristics of the deleted words that makes sufficient difference 
to warrant its general acceptance. 

Operatio nally defined cloze criterion score. 

The third aspect of the investigation was to detennine operationally 
a cloze criterion score for performance on material estimated to be suitable 
for children's independent or unsupervised reading. 

Table 20 shews the individual scores for each deletion pattern and 
the mean and standard deviation for each passage. 
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TABLE 20 

Individual scores f or each deletion pattern and mean and standard 

deviation for each passage. 



Deletion Pattern 



Passage 


1 


2 


3 


4 


5 


6 


7 


Mean 


S.D. 


School 1 




















Musical Seal 


26 


22 


26 


25 


20 


32 


23 


24.85 


3.56 


Paul Revere 


23 


19 


16 


29 


28 


22 


20 


22.42 


4.37 


Loaded Dog 


21 


1Q 




1 A 


1 ft 
1 O 


d\J 


^n 
J? 


0 A or 

21 .85 


7.39 


Aunt Letty 


8 


25 


13 


22 


1 1 
1 I 


>f 


J 1 * 


c. I • UU 


y. yi 


School 2 




















The Claimant 




70 

>o 




OA 


IO 


on 


on 
27 


29.00 


4.40 


The Smiths 


24 


22 


1? 




I D 




1 ft 
1 O 


■4 0 nr 

19.85 


4.36 


Insist 


16 


26 


22 


26 


26 


20 


23 


22.71 


3.49 


Prog Prince 


29 


29 


31 


34 


42 


28 


36 


32.71 


4.65 


Snhnnl o" 




















ILLgllaXlJ S oail 


J\ 




23 


29 


29 


24 


28 


27.57 


2.72 


r miuin inieves 




1 ft 
1 O 


23 


18 


21 


16 


19 


19.71 


2.49 


Paddin^rton "R 


1 




97 


JO 


do 


2p 


28 


29.57 


3.89 


Jack 


30 


21 


28 


21 


21 


19 


20 


22.86 


3.98 


School 4 




















Wouldn't Box 


29 


36 


36 


25 


34 


36 


33 


32.71 


3.92 


Christmas Ts. 


26 


22 


12 


20 


♦ 


25 


♦ 


21.00 


4.98 


Seal Eamily 


24 


18 


28 


43 


7 


19 


21 


22.86 


10.19 


Rip Van Winkle 


16 


24 


15 


22 


8 


9 


13 


15.29 


5.60 



* These two were not attempted due to the absence of the subject; 



An analysis of the individual scores shows a substantial range, with 
a low of 7 and a high of 43, both of which occurred in the same passage 
(School 4, passage 2). 

The mean scores for the passages range from a low of 15.29 (School 4, 
passage 4) to a high of 32.71 (School 2, passage 4, and School 4, passage 1). 
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The standard deviations for these passages are comparatively low and 
therefore it could be suggested that Rip Van Winkle was, relatively, a poor 
choice being too hard (not one of the seven subjects had a replacement rate 
of more than 48$, and Frog Prince and Wouldn't Box , relatively, too easy 
(all subjects having a replacement rate in excess of 5C#). 

Two of the passages showed relatively high standard deviations. For 
School 4, passage 3, ihe standard deviation was 10.19. This can be explained 
by the fact that two of the scores, 7 and 43, were grossly different, whereas 
the rest of the scores were very similar, in investigation of the deletions 
for these two patterns indicates that they were the easiest and most 
difficult for the passage, although the difference probably wouldn't account 
for the big difference in correct replacements. For School 1 , passage 4, 
the standard deviation was 9.91. Apart from the fact that pattern 1 was by 
far the most difficult - the subject only obtained a score of 8 - the marked 
variations in scores for this passage can probably only be accounted for by 
lack of homogeneity in the group. 

The overall mean replacement score of 24.07 gives an operationally 
defined score for cloze tests used in this investigation of 48.1 4556 with a 
standard deviation of 14.776. If these two figures are rounded off the 
results indicate a mean of 4S£ with a range of 33# - 63# (48 + 1 5) accounting 
for two thirds of the scores. 

Table 21 compares the results of this study with those of the earlier 
studies reported in Chapter 2. 

TABLE 21 I 

Cloze criterion scores for the Independent level . 

Bormuth 1967 50^ 

Bormuth 1968 57^ 

Rankin and Culhane 1969 61# 

Anderson and Hunt 1972 5356 

Boyce 1974 4^ 

Table 21 indicates that the score operationally obtained in this study is 
lower than the equivalent scores found in the previous studies, especially 
that of Rankin and Culhane. 
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This present study has a standard deviation associated with it however, 
whereas all the others provided single criterion scores. If it could be 
assumed that a s imilar standard deviation could be associated with the 
criterion scores previously reported, there is a range of scores from 
46# to 63# that is common to all students* (See Figure 2). 



Bormuth 1967 

Bormuth 1968 

Rankin and 
Culhane 1969 

Anderson and 
Hunt 1972 

Boyce 197I* 




irrr ' 



*"~ " ' }-| * "+ '* r "* 4-{-~ — .1.1 i 

j-j-'-r-' tJ t-H rn 1 1 ; !" "". 



ko% 



h6% 63% 

Cloze Replacement Score 
50* 60% 



10% 



Figure 2 . Common range of scores for independent level. 



Furthermore, if the standard deviation found in this present study- 
could be associated with the criterion scores for both the independent and 
instructional levels reported in earlier studies this would show whether it 
could be expected that any scores would fall into both of these levels. 

Figure 3 shows the information for all four previously reported studies 
showing the criterion scores obtained, together with the range associated 
with a standard deviation of 15, and the range for each study that is common 
to both reading levels. Table 22 summarizes this information. 
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Bormuth 1967 



Bormuth 1968 

Rankin and 
Culhane 1969 

Anderson & 
Hunt 1972 




20% 




4 
50* 



h4-"-.' 1 . 



i 

70JC 



6o* 



Cloze Replacement Score 



Fiffjre ?» Ran « e of s c «es associated with both the independent and 
instruction*! reading levels (See also Table 22)/ 



TABLE 22 

Range of scores associated with the 



independent 



and instructiona l levels of reading . 

Bormuth 1967 35 - 54 

Bormuth 1968 42 - 59 

Rankin and Culhane 1 969 46 - 56 

Anderson and Hunt 1 972 38 - 59 

In all cases the range of scores that might be expected to be associated 
with both levels of reading is large - the highest being 22 for Anderson and 
Hunt and the lowest being 11 for Kankin and Culhane. Whilst it is artificial 
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to associate the standard deviation from one study with the results of 
totally different studies, and whilst it may be said that the standard 
deviation in this present study might be rather high because of the material 
used and the possible lack of homogeneity in the members of the groups doing 
the tests, it is still likely that there would be a relatively large range 
of scores associated with both the independent and instructional levels of 
reading. 

Because of this it seems likely that the use of cloze scores associated 
simply with instructional and independent levels of reading may lead to 
rather gross judgements. It is probable that what is needed is a greater 
degree of differentiation in the material used to obtain appropriate cloze 
performance levels. For example, it would probably be more appropriate to 
obtain scores on a variety of levels such as material that is: 



and attempt to aim at the maximization of differences between the levels. 



(a) 
(b) 
(c) 
(d) 
(e) 
(f) 



much too difficult, 

rather difficult but with very high interest, 
eaay independent level judged by teacher, 
independent level judged by child, 
much too easy, 

instructional level judged by teacher, 
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CHAPTER VI 
SUMMARY AND CONCLPSIONR 

This investigation has sought to examine the measurement of passage 
performance by the use of the objectively determined cloze procedure. 

Previous attempts to measure passage performance using this procedure 
have involved the acceptance of the KLlgallon (1942) - Betts (1946) 
criteria for frustrations, instructional and impendent levels of 
reading, and have determined cloze scores comparable to the multiple- 
choice criteria for these levels (Bormuth 1967, 1968; Rankin and Culhane 
1969; Anderson and Hunt 1972). One attempt has been made (Bormuth 1 971 ) t 0 
determine criteria for passage performance using cloze criteria alone. 

All these previous studies have resulted in single cloze scores 
comparable to the 75# and 90* multiple-choice criteria, or in the case of 
Bormuth (1971), single cloze criteria scores for optimal efficiency 
according to the type of reading material and the grade level. 

This present study has attempted to inquire whether a single cloze 
criterion score can be misleading if it is being used to determine the 
suitability of reading material for an individual, as the score a child 
obtains appears to be a function of the types of words deleted. It is 
therefore feasible that for the possible cloze tests over any given passage 
there may be widely fluctuating levels of difficulty depending upon the 
actual combination of words being deleted in any one cloze deletion pattern. 

Using material said to be at the independent (unsupervised) level of 
reading for the subjects involved three matters were investigated. 
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1 . The characteristics of deleted words which make them easy or 
difficult to replace. It was found that the easiest words to replace were 
those that were 1-2 letters long, were one syllable long, were in common 
word lists and were articles, conjunctions, prepositions or pronouns. 

2. A means of adjusting the obtained scores to make allowance for 
the characteristics of the words deleted. 

For this purpose a regression analysis was used to determine formulas 
to adjust the actual score obtained by taking into account the difference 
between the score that would be predicted from the number of deleted words 
with certain characteristics and the mean criterion score. 

Two formulas were determined for computing the predicted score: 

(a) using the best single predictor (the number of words in common 
word lists), and 

(b) using the best pair of predictors (the number of words in common 
word lists and the number of words 1-4 letters long). 

Although the formulas derived would have been relatively easy for 
classroom teachers to use it was decided that the work associated with 
adjusting the scores was not justified in terms of the actual gain. 

Cooley (1971), quoting Burket 1964, Herzberg 1969, and Harks 1966, 
suggests that predictor weights often do not correlate as well with the 
criteria for new samples as they did with the original sample. He feels 
that in many cases "rather simple alternatives to regression weights, such 
as using the elements of r c directly (or even just unit weights. 1 ) 
frequently outperform the B weights on cross validation." (p.619) He 
claims that the problem diminishes as the number of predictors gets larger, 
at least 1 0 or 20 to 1 . In this investigation only five predictor 
variables were used. 

3. The determination of an operationally obtained cloze criterion 
score that could be used to determine whether materials is suitable for 
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the child f s unsupervised or independent reading. 

For this purpose the overall mean and standard deviation were obtained 
from 112 different cloze tests (seven different patterns of 16 different 
passages) used in this investigation. As the passages used for the cloze 
tests were from books deemed by the teacher to be suitable for unsupervised 
reading by the children involved, it should follow that the range given by 
the mean plus or minus one standard deviation should give an estimate of 
scores that could be expected to be achieved by two thirds of the children 
doing cloze tests from material suitable for independent reading. 

An analysis of the results obtained, together with comparisons made 
with the results in the earlier reported studies, suggests that there is 
likely to be an overlap between expected scores on the independent and 
instructional levels. 

The data gathered in this present study does highlight the weakness 
of a single criterion score as an indication of passage performance. Far 
more flexibility is required than is given by the oversimplification of 
all the factors implied in the use of a single criterion. 

Some limitations of the study . 

There are a number of factors that need to be considered in relation to 
the results of this investigation. 

In the first place the results are only really generalizable in terms 
of situations where:- 

(a) the subjects are sixth grade children; 

(b) there is a fifty item, one in seren deletion pattern; 

(c) the foiroat is a photocopy of the original passage with the 
deleted words whited out, and a separate answer sheet provided; 

(a) the children do a practice example first; 
(e) the same type of instructions are used. 
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Generalizing the results outside this fairly rigid set of constraints 
would not be really justified. There is insufficient evidence in the 
literature regarding the equivalence of cloze performance at various grade 
levels, using different n^ word deletions over the same material, or using 
different test formats, instructions and strategies. So, for example, it 
could not be said that the results obtained in this stucly would be 
applicable to an every fifth word deletion, using a "typed format where the 
children write the answer in, where the children are instructed to read 
through the passage to grasp meaning before attempting to replace any words, 
and where the subjects are fifth or fourth grade children. In fact, a 
change in any one of these factors might well affect the valid use of the 
results from this study. There appears to be great scope for a large 
number of different studies to explore the relationships between factors 
such as these and the obtained scores. 

Secondly, there is the possibility that the materials used for the 
tests in the main investigation were not uniformly at the independent level 
of reading for all the children involved. Although the teachers were asked 
to supply books to meet this criterion, it is possible, or even probable, 
that the books they chose were directed more to the mean for each group of 
seven rather than for the group as a whole. This would probably have had 
little effect, if each group was fairly homogeneous in ability, but it 
could be argued, particularly for each group 1 (thebest readers) and each 
group 4 (the poorest readers), that the range of ability may have been 
quite wide. Thus, although this may have balanced out and the effect on 
the obtained mean score been small, it may have resulted in a larger 
standard deviation. However, as indicated earlier, the poorest readers in 
each grade were not included, and a check on the groups in the grades where 
teachers did have reading test scores available indicates that the groups 
were relatively homogeneous. 

It may have been better, though not very practical, to have chosen a 
book specifically for each child, and had each child do - at reasonable 
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intervals - all seven patterns for the chosen passage* Another alternative 
may have been to have ranked all the children in the four grades according 
to a standardized test of reading comprehension such as the Schonell R4 f 
the Neale Analysis of Reading Ability, or Daniels and Diaks Test 10. 
Groups of seven could then have been deteimimd over all four grades by 
matching scores as closely as possible. 0!he choice of book for each group 
would then have required the concensus of opinion of the four teachers 
involved. 

A more serious problem is the fact that all the books chosen were in 
fact anthologies or collections of stories. Because of this the possibility 
exists that there were wide fluctuations in the difficulty levels of the 
various stories contained in any one book. Although the passage used from 
any one book was chosen at random, and this randomizing process may have 
evened out the possible differences in difficulty, the possibility still 
exists that some of the passages chosen were not representative of the 
overall difficulty of the book. It is probable that it would have been 
better to have used more than one passage from the book, to have used, for 
example, three passages each 120 words lon</> rather than only one 350 word 
passage. Possibly an eveji better solution wculd have been to have asked 
the teachers to choose a passage from a book rather than simply a book. 
In this way there would be more certainty in the belief that the passages 
chosen were, in the teachers 1 opinions, representative of the independent 
level of reading for the children involved. 

Finally, in relation to this second problem is the question raised 
earlier of the reliability of teacher estimates of the suitability of 
materials. As was quoted earlier, Klare (1963, p.81) states that "they 
are recognized as subject to considerable error". The teachers chosen for 
this study were all veiy experienced teachers, with excellent records as 
teachers, and with a particular interest in reading. All had taught the 
children used as subjects in this investigation for twelve months and 
should therefore have had a good understanding of the ability of each 
individual in the area of reading, and more specifically, reading 
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comprehension. Notwithstanding this, however, there must be some doubt a 
to the aptness of the choice of books made for the particular subjects 
involved, although it is hoped that the particular choice of teachers has 
minimized this doubt. 



A third factor that needs to be considered is the method used for 
scoring the responses. The rationale for the exact replacement method 
used in this investigation has been dealt with earlier in Chapter 2. There 
must be some doubt however as to whether this is the most appropriate method 
of scoring to use for this particular purpose. Whilst it is acknowledged 
that it is the only truly objective method of scoring, and that it is far 
simpler, the fact that the test is being used to determine whether the 
material is suitable for an individual must be considered. It is possible 
that there are a number of different types of replacement that might serve 
as indicators that the individual is understanding or comprehending the 
material in the passage. These different types of replacements include not 
only similes and logical replacements, but also the use of basically the 
correct word but the wrong number (e.g. man instead of men) or the wrong 
tense (e.g. runs instead of ran). It is also possible that on some 
occasions the use of the correct part of speech, even if the word is quite 
wrong, may indicate a grasp of the material. The concepts of restricted 
and elaborated codes of language may well be pertinent to this question. 
It is possible that restricted language users might comprehend the 
language in the passage but that their own usage might prevent them from 
replacing certain words correctly, thus leading to artificially low 
estimates of their comprehension of the material. This could be a 
possibility for children from lower class or ethnic minority backgrounds. 
It is also possible that elaborated code users may use enriched vocabulary 
in some cases, only to find that these are scored as incorrect. The recent 
work of Poole (1973) is very relevant to this question of social class 
differences in language and the cloze procedure. 

It is therefore possible that one of the major claims to simplicity in 
the cloze procedure, the scoring of only exact replacements, may work to the 
disadvantage of some children. 
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Fourthly, there appears to have been no study of the effects of the 
cloze procedure on the motivation of the children to succeed. 

An investigation of the results in the Pilot Study (Chapter 3) 
indicates that for the first 20$ of deletions, the subjects doing pattern 4 
(easy) had a 57.0^ correct replacement rate, whereas for those doing 
pattern 5 (hard) the rate was only 31. 1#. From that point on the subjects 
doing pattern 5 actually did better as the final correct replacement rates 
were 50.8($ for pattern 4 (down 6.2#) and 32.79# for pattern 5 (up 1.68$. 

Although the opposite appeared to happen in this case, it could be 
speculated that if the first part of a cloze test contains a high number of 
difficult deletions this might have a depressing effect on 1he subjects. 
This could be tested by having half the subjects do the 'easy' version of a 
cloze test and the other half do the first 2<# of the hard version before 
adjusting it to make the final 80^ the 'easy 1 version. The mean scores 
for the final 80^ could then be compared. 

It is possible that difficult to replace words at the beginning of a 
passage do not matter too much if the subjects see their task simply as one 
of seeing how many gaps they can fill in, thus approaching the task from a 
'bit 1 rather than a 'whole 1 approach. The usual instruction "You may skip 
hard blanks and come back to them again" may well reinforce a strategy of 
not being very concerned about the total context. As was mentioned earlier 
(Chapter z) Boyce (1972) found that many subjects filled in blanks with 
words which were logical in the immediate context, but which were incorrect 
or even illogical in the total context of the passage. Shis seems to 
suggest a 'bit 1 approach. 
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Conclusion 

This thesis has explored the use of the cloze procedure as a means of 
determining, by direct testing, the suitability of written material for 
the individual, and in particular, for his independent, unsupervised 
reading. 

The strengths of the cloze procedure for this purpose lie in its basic 
simplicity, its objectivity, and its ability to match the child's 
performance with the actual material. Its basic weakness lies in 
problems associated with the meaningful interpretation of the score 
achieved by an individual as a result of cloze tests on the material. 

Attempts have been made (Boimuth, 1967, 1968; B^lr jn and Culhane, 1969; 
Anderson and Hunt, 1972) to relate scores on cloze tests to scores on 
multiple-choice tests of the same material and then use these comparable 
scores as criteria for interpreting the individual's performance, thus 
determining the suitability of the material for him. This thesis has 
explored this approach and has shown that there are problems associated 
with it, problems which cast seme doubt on the effectiveness of the 
cloze procedure for this purpose. 

It should be noted however that these doubts about the effectiveness of 
the cloze procedure apply only to the interpretation of cloze scores 
for passage performance purposes. The findings of the investigations 
reported in the thesis do not imply criticism of the procedure for many 
of the other purposes for which it is used. For many purposes the cloze 
procedure appears to be an exceptionally robust and useful measure. 
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APPENDIX A 

SUGGESTIONS TOR THE WHITING OP MULTIPLE 
CHOICE TEST ITEMS 
(Vesman, 1971 ) 

General 

1 . The item writer must have a thorough mastery of the subject matter 
being tested. Not only must he be acquainted with the facts and 
principles of the field he must be fully aware of their implications. 

2. The writer who prepares items for use in tests of educational achievement 
must possess a rational and well-developed set of educational values 
(aims or objectives) that so permeate his thinking that he tenia 
continually to seek these values in all his educational efforts. 

3. Die item writer must understand psychologically and educationally the 
individuals for idiom the test is intended. 

4. The item writer must be a master of verbal communication. 

5. The item writer must be skilled in the handling of the special 
techniques of item writing. 

6. As item writing is not a unitary skill, the item writer must be adept at 
writing the appropriate types of items for the subject matter being 
tested. 

General suggestions for* vH-Hyi ff objective items ; 

1 . Express the item as clearly as possible. 

2. Wherever possible, choose words that have precise meanings. 

3. Avoid complex or sideward word arrangements. 

4. Include all qualifications needed to provide a reasonable basis far 
response selection. 
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5. Avoid the inclusion of nonfunctional words. 

6. Avoid unessential specificity in the stem or the responses. 

7. Be as accurate as possible in all parts of an item. 

8 # Adapt the level of difficulty of the item to the group and purpose for 

which it is intended. 
9. Avoid irrelevant clues to the correct response. 
10 # Avoid stereotyped phraseology in the stem or the correct response. 

1 1 . Avoid irrelevant sources of difficulty. 

12. Expose items to expert editorial scrutiny. 

Specific to multiple-choice items. 

1 . Use either a direct question or an incomplete statement as the stem. 

2. In general, include in the stem any words that otherwise must be 
repeated in each response. 

3. Avoid negatively expressed stems if possible. 

4. Provide a response that competent critics can agree on as best. 

5. Make all the responses appropriate to the item stem. 

6. Make all dis tractors plausible and attractive to examinees who lack 
the information or ability tested by the item. 

7. Avoid highly technical dis tractors. 

8. Avoid responses which overlap or include each other. 

9. Use 'none of these 1 as a response only in items to which an absolutely 
correct answer can be given. 

1 0 # Arrange the responses in logical order, if one exists, but avoid a 
consistent preference for any particular response po&ition. 

11. If the item deals with the definition of a term, it is usually preferable 
to include the teim to be defined in the stem. 

12. Do not present a series of true-false statements as a multiple-choice 
item. 
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APPENDIX B 



LIST OP 'KEY 1 . 


'BASIC 1 . 'INSTANT 1 


AND 'SIGHT' WORDS 




A 


been 


children 


end 


about 


before 


Christmas 


enough 


aeroplane 


being 


city 


even 


after 


best 


clean 


eveiy 


again 


better 


close 




against 


between 


colour 


fact 


all 


big 


come 


far 


almost 


bird 


could 


fast 


also 


birthday 


course 


father 


always 


black 


cowboy 


fell 


am 


blue 


cut 


few 


an 


bock 




field 


and 


boat 


daddy 


find 


another 


both 


day 


fine 


any 


bought 


dear 


fire 


are 


box 


did 


first 


around 


boy 


dinner 


fish 


as 


bring 


didn't 


five 


ask 


brother 


do 


flower 


at 


brought 


does 


fly 


auntie 


but 


dog 


for 


away 


buy 


doll 


found 




by 


don't 


four 


baby 




door 


friend 


back 


call 


down 


from 


bad 


came 


dress 




ball 


camp 


during 


game 


be 


can 




garden 


because 


car 


each 


gave 


bed 


cat 


eat 


general 
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get 


I 


girl 


if 


give 


in 


glad 


into 


go 


is 


going 


its 


good 




got 


jump 


grandma 


just 


great 




green 


keep 



had 

hand 

has 

have 

he 

head 

help 

her 

here 

high 

him 

himself 

his 

home 

hope 

horse 

house 

how 

however 



kind 
know 

large 

last 

leave 

left 

less 

let 

letter 

life 

like 

little 

live 

long 

look 

lot 

made 
make 



man 

many 

may 

me 

men 

might 

more 

morning 

most 

mother 

Mr 

Mrs. 

much 

ffiunmy 

must 

hqt 

name 

near 

never 

new 

next 

night 

nice 

no 

not 

nothing 

now 

number 

of 



off 

old 

on 

once 

one 

only 

open 

or 

other 

our 

out 

over 

own 

paper 

part 

Party 

people 

pick 

picture 

place 

play 

please 

present 

pretty 

public 

put 

rabbit 

ran 

read 
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red 

right 

room 

round 

run 



said 

same 

sat 

saw 

say 

school 

see 

seen 

set 

shall 

she 

ship 

shop 

should 

side 

since 

sing 

sister 

sit 

sleep 

small 

snow 

so 

some 

something 
soon 



stand 
start 
state 
still 
stop 
story- 
such 
summer 
sure 
system 

take 



teacher 

television 

tell 

than 

that 

the 

their 

them 

then 

there 

these 

they 

thing 

thick 

this 

those 

though 

thought 

three 

through 
time 



to 

today 

told 

tonight 

too 

took 

train 

two 

tree 

under 

until 

up 

upon 

us 

use 

yery 

walk 
want 
war 
was 

watch 
water 
way 
we 

week 
well 
went 
were 
what 
when 
where 



which 

while 

white 

who 

why 

will 

wish 

with 

without 

woman 

wozk 

would 

write 

year 
yes 

y*t 

yesterday 

you 
your 
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APPENDIX C 



PERCENTAGE 


REPLACEMENT RATES 


FOR EACH OF 


THE 1 




WORDS 


IN THE CLARK AMD JOHNSON 1972 


STUDI. 




from 


AC A 

45.4 


lire 




o70 


ridden 


OC A 

25 #4 


lonely 




C A 

5»4 


or 


OC A 

0#4 


tnere 




a\ n 
oi «o 


taught 




from 






knew 


AC A 

45#4 


Spring 




5»4 


holes 


CC A 

o5.4 


the 




1 o # 4 


few 


CA C 

54#5 


or 




1A K 

74«5 


the 


7^.9 


are 




70 


for 


OI ft 


their 




ou.u 


holes 


CO ^ 

52.7 


a 




ou.u 


that 




tor 




OQ 4 

29.1 


another 


Cft ft 


own 




OQ 4 


for 


2U«9 


at 






of 


Of #p 


to 




oi «o 


each 


o^ a 


be 




HA C 


Doug 


OA ft 


he 






rough 


7.3 


better 






he 


7U8 


first 




12.7 


small 


15.4 


good 




61 .8 


tree 


60.0 


with 




41 .8 


a 


27.5 


forehead 


18.2 


section 


3.6 


him 




70.9 


enough 


32.7 


but 




56.4 


partially 


7.3 


a 




45.4 


taught 


10.9 


between 




25.4 
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APPENDIX D 

SAMPLE OF CLOZE TEST USED FOR PILOT STUDY 
CLOZE TEST B (Pattern 5 ) 

DOUS OF AUSTRALIA 

Fran the veiy first 1 when he had 1 

ridden out to 2 range with his father 2 

or one 3 the stockmen, Doug had been 3 

taught 4 of surviving in the bush. 4 

He 5 the locations of half a dozen 5 

6 holes in the nearby foothills. 6 

After 7 a few more twigs on the 7 

8 to insure the calf's safety, he 8 

9 off to look for a drink. 9 

1 0 . few weeks ago the holes had 1 0 

1 1 m a little water. Doug discovered 1 1 

that 1 2 they were quite dry* One 1 2 

after 13 , he lifted rocks which 13 

served as 14 for the holes, but 14 

only a 15 film of dampness was left 15 

at 16 bottom of each hole. This 16 

was 17 but not disastrous. Doug 17 

rested on 18 granite face of the 18 

rough foothill 1 9 thought for a 19 

moment. Then he 20 back towards his 20 

fire and the 21 calf. He stopped 21 

erJc 13 



at a young 22 tree which was growing 

along the 23 . With a knife he 

managed to 24 off a section of root 

with 25 hollow heart containing 

enough water to 26 m his thirst at 

least partially. Long 27 Bex the 

aboriginal stockman, had told 28 

that this tree could be a 29 saver 

to a man lost in 30 lonely bush, 

and Doug had never 31 it. 

There were so many things 32 had 

learned from him over the 33 m Why, 

only last Spring Bex had 34 him how 

to throw the sharpened, 35 boomerang 

the aborigines used instead of 36 

gun to kill the animls that 37 

found in the bush. The aborigines 38 



their own boomerangs, whittling them out 

39 a special type of wood, after 

40 it for suppleness and strength. 

Dou « 41 at his own daydreaming 

because he 42 left his boomerang 

at home, anyway. 43 any case he had 

to confess 44 no wild animals seemed 

to be 45 around to provide a meal. 
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Perhaps 46 could find some witchetty 46 

grubs, which 47 better than nothing. 47 

It was Hex 43 had first explained to 43 

Doug that 49 grubs were good to eat. . 49 

Doug 50 see him now, with his old 50 



straw hat and his jutting forehead, and he wished 
the stockman were with him now. 



(Note: The tests used in the Pilot Study were in a foolscap format thus 
the first 31 deletions were on the front of the sheet followed by 
"Please Turn Over the Page and Continue M . ) 
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APPENDIX E 

WRITTEN INSTRUCTIONS FOR EXPERIMENTERS ADMINISTERING 
THE PILOT STUDY CLOZE TESTS 

The purpose of this investigation is to gather information about 
problems associated with the use of the cloze procedure as a measure of 
reading comprehension. 

1 . Show the material to the class teacher and discuss the purpose of 
the testing. If the class teacher gives approval to give the tests 
then: 

2. Hand out a copy of the test to each member of the grade according to 
normal classroom seating. There are, in fact, four different fonns 
of the test and they have been placed in the envelope in groups of 
four. When you hand out the tests please ensure that you han d them 
out in this order. This will ensure that no person will have the 
same test, or a test from the same material, as the person beside them. 

3. Read out the instruction page. Answer any questions in terms of what 
is in the instructions. 

4. Give the children five minutes to do the sample exercise. When they 
have finished, briefly go over the answers explaining why these 
particular words were the correct replacements. 

5. Ask the children to turn over the page and commence the main task. Do 
not directly answer any question that will give any clue to a missing 
word. 

6. Allow the children 25 minutes to complete the task. 

7. Collect all test sheets and place them back in the envelope. 

8. Hand the envelope to the member of staff who comes out to the school 
from the College and ask him/her to return it to me . 

ERJC lu 



If possible I would like you to do this in the first or second week 
of the teaching round. 



Thank you for your assistance. 
M.W. Boyce. 



Note: The nine student teachers who administered the tests for the 
Pilot Study were all volunteers, and had had a briefing session at 
College regarding the purpose of the testing before they took the 
tests out to the schools. All the class teachers who were asked 
co-operated. 
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APPENDIX F 

LIST OF SO ORCES BOB THE PASSAGES USED FOR THE CLOZE TESTS 
IN THE MAIN INVESTIGATION. 



School 1 
Group 1 : 

Group 2; 

Group 3; 
Group 4: 



The Musical Seal. R. Farre. In High Spirits. Education Depart- 
ment of Victoria, n.d. , pp 66-67. 

F&ul Revere and the World he lived in. E. Forbes. In Wagner G.W., 
and Wilcox L.A. and Persons G.L. (Eds) Readers Digest Reading 
Skill Builder, Readers Digest Services , 1959, PP 135-136. 

The Loaded Dog. H. Lawson. In The Victorian Reader Sixth Book . 
Education Department of Victoria, n.d., pp 154-155. 

How Aunt tetty Killed the Panther. (Anon. ) In The Victorian 
Reader Fifth Book , n.d., pp 131-132. " 



School 2 
Group 1 : 

Group 2: 

Group 3: 

Group 4: 



The Claimant. In Elowerdew, P. and Stewart, S. Beading Oat 
Yellow Book 2, Oliver and Boyd, 1963. pp 1 22-1 23" 

Meet the Sniths. In Elowerdew, P. and Stewart, S. Reading On; 
Red Book 1 , Oliver and Boyd, 1966. pp 80-61. 



A Question of Insight. J.B. Mosley. In New Reading Skill 
Builder ; Bart 1. Readers Digest, 1968, pp 20-21. 

The Frog Prince. The Brothers Grim. In Huber, M.B. and 
Salisbury, F.S. Magic Everywhere . James Nisbet and Co., 1962. 
PP 8-9. 



School 3 
Group 1 ; 



Afghanistan; Domain of the fierce and free. James A. MLchener. 
In Scott A.P. (Ed. ) New Reading; Red Book Eve. Readers Digest 
Educational Department. 1960, pp 30-31. 



Group 2: The Puddin' Thieves. Norman Lindsay. In The Planet of the Bees 
and Other Stories. Endeavour Beading Program 13. Jacaranda Press. 
1972, pp 8-9. 

Group 3; Goings On at No. 32. Michael Bond. In Reading for Pleasure , 
Endeavour Reading Program 11. Jacaranda Press, 1972. pp 62-63. 
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Group 4: Jack the story of a pretty good donkey. P.P. Jay. In Hew 

Reading Skill Builder Part ? Sinclair, K.M. and Sparks7~N.J. . 
Readers Digest, 1971, pp 86-87. 



School 4 

Group 1 : The Boy who wouldn't Box. In Lamb, G.P. One Hundred Good 
Stories . 

Group 2: Christmas Trees. In Flowerdew P. and Stewart S. Reading O n; 

Red Book 2. Oliver and Boyd, 1971. pp 8-9. 

Group 3: The Seal Family . In Schonell, P.J., Plowerdew, P., and Blliott- 
Cannon^A. Wide Range Interest; Book g Oliver and Boyd, 1971, 

Group 4: Rip Van Winkle. In Jack and the Stolen Amies , fioyal Road 
Readers Book 8. Daniels J.C. and Diak H. Chatto and Windus, 
1970, pp 30-32. ' 
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APPENDIX G 

SAMPLE OF THE CATEGORIZATION OF THE SEVEN CLOZE 
PATTERNS OF ONE OF THE SIXTEEN PASSAGES . 
The Musical Seal 



De le ti on Dele tion 



Pattern 1 


No 


No 


In 


P/P 


Pattern 2 


No 




Tn 

XII 


p/p 




its 


syl. 


Com 


c/a 




its 


syl 


Com 


C/A 


Lora's* 


5/6 


2 






musical * 


7+ 


3 






Aunt 


3/4 


1 


+ 




Miriam 


5/6 


3 






the * 


3/4 


1 


+ 


+ 


piano* 


5/6 


3 






no * 


1/2 


1 


+ 




notice 


5/6 


2 






Tfriggle 


7+ 


2 






over 


3/4 


2 


+ 


+ 


it * 


1/2 


1 


+ 


+ 


or 


1/2 


1 


+ 


+ 


and 


3/4 


1 


+ 


+ 


listen 


5/6 


2 






concentration 


7+ 


3+ 






and * 


3/4 


1 


+ 


+ 


swaying 


7+ 


2 






now 


3/4 


1 


+ 




body* 


3/4 


2 






to 


1/2 


1 


+ 


+ 


stopped * 


7+ 


2 






she* 


3/4 


1 


+ 


+ 


minutes* 


7+ 


2 






still 


5/6 


1 


+ 




to 


1/2 


1 


+ 


+ 


my* 


1/2 


1 


+ 


+ 


described 


7+ 


3 






as * 


1/2 


1 


+ 




me * 


1/2 




+ 


+ 


a* 


1/2 


1 


+ 


+ 


of * 


1/2 




+ 


+ 


songs 


5/6 


1 






through * 


7+ 




+ 


+ 


the 


3/4 


1 


+ 


+ 


would* 


5/6 




+ 




do* 


1/2 


1 


+ 




day * 


3/4 




+ 




For 


3/4 


1 


+ 


+ 


a * 


1/2 




+ 


+ 


time 


3/4 


1 


+ 




wild 


3/4 








raspberries 


7+ 


3 






animal 


5/6 








within 


5/6 


2 




+ 


or * 


1/2 




+ 


+ 


two* 


3/4 


1 


+ 




of * 


1/2 




+ 


+ 


Harlech 


7+ 


2 






a * 


1/2 




+ 


+ 


loud* 


3/4 


1 
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Deletion 
Pattern 1 



Deletion 
Pattern 2 





No 


Ho 


In 


P/P 




No 


No 


In 


P/P 




± bS 


syl 


Com 


n /a 




its 


syl 


Com 




I * 


i/p 


i 


+ 


+ 


saw* 


_ / 

3/4 


1 


+ 




she * 




i 
1 


1 

T 


+ 


broke 


5/6 


2 






"npT»'hciT)ci 

Jf ^ J» IXcXJ^O 


f + 








the* 


3/4 


1 


+ 


+ 


Their 


-V4 


A 
1 


+ 


+ 


repertoire 


7+ 


3 






mewing 


5/6 


O 
<— 






hisses 


5/6 


2 






rises 


5/6 


0 






from 


3/4 


1 






treble 


^/6 


0 






The 


3/4 


1 


+ 


+ 


I* 


1/p 


A 
I 


+ 


+ 


still 


5/6 


1 






reedy 


5/6 


O 






efforts 


7+ 


2 






had 


3A 

-«V » 


1 


4. 

1 




xne 


3/4 


1 


+ 


+ 


on 


l/p 


1 


+ 


+ 


he le- 


3/4 


1 


+ 


+ 


the* 


J/ ** 


I 


1 

T 


+ 


prae tic e 


7+ 


2 






played* 


5/6 


O 
C. 






a* 


1/2 


1 


+ 


+ 


slow* 




A 
I 






pace 


3/4 


1 






and* 


i/a 


I 


1 

T 


+ 


descending 


7+ 


3+ 






to* 


1/P 


I 


+ 


+ 


follow 


5/6 


2 






wail 




1 
1 






A •¥- 

A* 


1/2 


1 


+ 


+ 


or* 


1/2 


1 
1 


T 


T 


a* 


1/2 


1 


+ 


+ 


annoyed 


7+ 








ner* 


3/4 


1 


+ 


+ 


grunt* 


5/6 


1 






ana* 


'z/a 

3/4 


1 


+ 


+ 


flippers 


7+ 


2 






a* 


1/2 


1 


+ 


+ 


within 


5/6 


2 




+ 


a* 


1/2 


1 


+ 


+ 


get 


3/4 


1 


+ 




through 


7+ 


1 


+ 


+ 


Danny 


5/6 


2 






Boy* 


3/4 


1 


+ 




beginning 


7+ 


3+ 






to* 


1/2 


1 


+ 


+ 
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The Musical Seal 



Deletion 
Pattern 3 



Deletion 







Mn 
1MU 


Tn 
XII 


p/p 

'1 




WO 


NO 


In 






Its 


syl 


Com 


elk 




Its 




Com 




talent 


5/6 


2 






came 


3/4 








or 






1 

T 


1 

T 


I* 


1/2 




+ 


+ 


the 






4. 


4. 
1 


other * 


5/6 




1 

T 




11 U 0 






4* 
T 


1 

*t* 


so 


1/2 




+ 




to 


1/2 




1 


4. 


the* 


3/4 




+ 


+ 


more 






4* 




inconveniently 7+ 








with * 


3/4 


1 


+ 


+ 


an* 


1/2 




4* 


4* 


i o v 


3A 


1 






which * 


5/6 




4* 
1 


4> 
T 


dilU. 






4- 


4. 
1 


then * 


3/4 




+ 




the * 


3/4. 




+ 


+ 


music 


5/6 


0 






w uuiu 


2/o 




4> 

T 




sit 


3/4 




1 

T 






2/ O 




4. 


4* 


its 


3/4 




1 

T 


1 

T 


o *i ncri Tier 


7+ 


0 






however 


7+ 








"hniTrn T t a "M nrr 


7+ 








A 


1/2 




1 

T 


.1 


rn nfi i "hVi "X" 
m uu uxx 


2/ O 








organ * 


5/6 








for * 






+ 
1 


4. 

1 


a* 


1/2 




1 

T 


1 

T 


book 


3/4 


1 


+ 




I* 

JL 


1/2 




4- 


4* 
1 


a * 


1/2 


1 


+ 
1 


4- 


little 


5/6 




4* 




the * 






+ 
1 


4* 

r 


first 


5/6 




4. 
T 




when * 


3/4 


1 


+ 


4* 


Aunt 


3/4 




1 

T 




wild * 


3/4 


1 


+ 
1 


4* 


there* 


5/6 




4. 
T 




sight 


5/6 








After 


5/6 




4. 
T 


1 

T 


I * 


1/2 




+ 


4* 


started* 


7+ 








To * 


1/2 




+ 
1 


4* 
1 


my* 


1/2 




1 

T 


1 

T 


51 UctXI 


O/O 








beside 


5/6 






+ 


Lora * 


3A 


2 






and* 


3/4 




+ 


+ 


into * 


3/4 


2 


+ 


+ 


a* 


1/2 




+ 


+ 


largest* 


7+ 


2 






vocal 


5/6 








includes 


7+ 


3 






grunts 


5/6 








and * 


3/4 


1 


+ 


+ 


a 


1/2 




+ 


+ 



o 
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Deletion 
Pattern ^ 



a * 


No 
Its 

1/2 


No 
syl 

■1 ■ ■ 

1 


In 
Com 

+ 


P/P 

c/a 

+ 


roar 


3/4 


1 






took* 


3/4 


1 


+ 




were* 


3/4 


1 


+ 




idea 


3/4 


2 






own 


3/4 


1 


+ 




sessions 


7+ 


2 






simple 


5/6 


2 






with 


3/4 


1 


+ 


+ 


notes 


5/6 


1 






the* 


3/4 


1 


+ 


+ 


sudden 


5/6 


2 






piece 


5/6 


1 






for 


3/4 


1 


+ 


+ 


beat 


3/4 


1 






habit 


5/6 


2 






week 


3/4 


1 






Baa* 


3/4 


1 






without* 


7+ 


2 


+ 


+ 


learn* 


5/6 


1 







Deletion 
Pattern 4 





No 


No 


In 






Its 


sjrl 


Com 




deep 


•z /a 

3/4 


1 






turned 


1- //- 

5/6 


2 






no* 


1/2 


1 


+ 




soon 


T / A 

3/4 


1 


+ 




ox ^ 


1/2 


1 


+ 


+ 




1/2 


1 


+ 


+ 




3/4 


1 


+ 


+ 


tune 


•Z 

3/4 


1 






bars 


3/4 


1 






she 


■Z />! 

3/4 


1 


+ 


+ 


music 


5/6 










■z/ii 

3/4 


1 


+ 




pxayea* 


r- //- 

5/6 










3/4 


1 


+ 


+ 


about 


5/6 




+ 




of* 


1/2 




+ 


+ 


she* 


3/4 




+ 


+ 


Baa* 


3/4 








a* 


1/2 




+ 


+ 


where* 


5/6 




+ 
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The Musical Seal 



Deletion 
Pattern 5 



No No In P/p 
Its s^l Com C/A 



Deletion 
Pattern 6 



No 
Its 



No In P/P 
sjrl Cpm C^A, 



out 


3/4 


1 


+ 




early 


5/6 


2 






struck 


5/6 


1 






up 


1/2 


1 


+ 


+ 


animals 


7+ 


3+ 






would* 


5/6 


1 


+ 




Lora 


3/4 


2 






she* 


3/4 


1 


+ 


+ 


instrument 


7+ 


3+ 






lean 


3/4 


1 






the* 


3/4 


1 




+ 


player's 


7+ 


2 






expression 


7+ 


3+ 






of* 


1/2 


1 


+ 


+ 


was* 


3/4 


1 


+ 




quite 


5/6 


1 






with* 


3/4 


1 


+ 


+ 


her* 


3/4 


1 


+ 


+ 


when* 


3/4 


1 


+ 


+ 


the* 


3/4 


1 


+ 


+ 


quietly 


7+ 


3+ 






for* 


3/4 


1 


+ 


+ 


spell* 


5/6 


1 






Her* 


3/4 


1 


+ 


+ 


can 


3/4 


1 


+ 




only 


3/4 


2 


+ 




relation 


7+ 


3+ 






had* 


3/4 


1 


+ 




and* 


3/4 


1 


+ 


+ 


a* 


1/2 


1 


+ 


+ 


birthday 


7+ 


2 






present* 


7+ 


2 






decided 


7+ 


3+ 






that* 


3/4 


1 


+ 


+ 


singing 


7+ 


2 






practice 


7+ 


2 






session 


7+ 


2 






I* 


1/2 


1 


+ 


+ 


was* 


3/4 


1 


+ 




out* 


3/4 


1 


+ 




was* 


3/4 


1 


+ 




not* 


3/4 


1 


+ 




a* 


1/2 


1 


+ 


+ 


preliminary 


7+ 


3+ 






off 


3/4 


1 


+ 


+ 


on 


1/2 


1 


+ 


+ 


annoyance 


7+ 


3+ 






I* 


1/2 


1 


+ 


+ 


me* 


1/2 


1 


+ 


+ 


Looking* 


7+ 


2 






continued 


7+ 


3+ 






singing* 


7+ 


2 






roar 


3/4 


1 






seals* 


5/6 


1 






range 


5/6 


1 






among 


5/6 


2 






snorts 


5/6 


1 






barks 


5/6 


1 
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Deletion Deletion 
Pattern 5 Pattern 6 





No 


No 


In 


P/P 




No 


No 


In 


P/P 




Its 

JL bo 




Horn 


P/A 




its 




Com 


C/A 


Wail 


3/4 


1 






which* 


5/6 


1 


+ 


+ 


bass* 


3/4 


1 






to* 


1/2 


1 


+ 


+ 


to 


1/2 


1 


+ 


+ 


a* 


1/2 


1 


+ 


+ 


notice 


5/6 


2 






but 


3/4 


1 


+ 


+ 


outclassed 


7+ 


3+ 






then* 


3/4 


1 


+ 




letting* 


5/6 


2 






her 


3/4 


1 


+ 


+ 


my 


1/2 


1 


+ 


+ 


accompaniment 


7+ 


3+ 






followed 


7+ 


3+ 






when* 


3/4 


1 


+ 




at* 


1/2 


1 


+ 


+ 


a* 


1/2 


1 


+ 


+ 


of 


1/2 


1 


+ 


+ 


steadily 


7+ 


3+ 






made* 


3/4 


1 


+ 




valiant 


7+ 


3+ 






in 


1/2 


1 


+ 


+ 


a* 


1/2 


1 


+ 


+ 


or 


1/2 


1 


+ 


+ 


low* 


3/4 


1 






too 


3/4 


1 


+ 




quickly* 


7+ 








would* 


5/6 


1 


+ 




start 


5/6 


1 


+ 




with* 


3/4 


1 


+ 


+ 


her* 


3/4 


1 


+ 


+ 


hers* 


3/4 




+ 


+ 


when* 


3/4 




+ 




was* 


3/4 




+ 




able* 


3/4 








Black* 


5/6 




+ 




sheep* 


5/6 








break 


5/6 








and* 


3/4 




+ 


+ 


my 


1/2 




+ 


+ 


Caravan 


7+ 


3+ 
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The Musical Seal 



Deletion 
Pattern 7 



Deletion 



No Wo In p/p 
Its syl Com c/a 



whenever 


7+ 


3+ 




on* 


1/2 


1 


+ 


take* 


3/4 


1 


+ 


would* 


5/6 


1 


+ 


against 


7+ 


2 


+ 


legs 


3/4 


1 




intense 


7+ 


2 




flattering 


7+ 


3+ 




whole 


5/6 


1 




music* 


5/6 


2 




several 


7+ 


3+ 




reactions 


7+ 


3+ 




be* 


1/2 


1 


+ 


sent* 


3/4 


1 




book* 


3/4 


1 


+ 


Thumbing 


7+ 






I* 


1/2 


1 


+ 


each* 


* 

3/4 


1 


+ 


chose 


5/6 


1 




picking* 


7+ 






an* 


1/2 


1 


+ 


scale 


5/6 






Men 


3/4 




+ 


heard* 


5/6 






down* 


3/4 




+ 


Whereupon 


7+ 


3+ 




have* 


3/4 


1 


+ 


mammals 


7+ 


2 




peculiar 


7+ 


3+ 





Pattern 7 


Wo 
Its 


Wo 


In 

Horn 


P/P 

p/a 


often* 


5/6 


2 






a* 


1/2 


1 


+ 


+ 


hiss 


3/4 


1 






my 


1/2 


1 


+ 


+ 


I* 


1/2 


1 


+ 


+ 


sing* 


3/4 


1 


+ 




During 


5/6 


2 






I* 


1/2 


1 


+ 


+ 


fairly- 


5/6 


2 






ascending 


7+ 


3+ 






efforts 


7+ 


2 






timeless 


7+ 


2 






note* 


3/4 


1 






plainly 


7+ 


2 






to 


1/2 


1 


+ 


+ 


fore 


3/4 


1 






angry 


5/6 


2 






to* 


1/2 


1 


+ 


+ 


and* 


3/4 


1 


+ 


+ 


was* 


3/4 


1 


+ 




has 


3/4 


1 


+ 
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APPENDIX H 



Samples of Cloze Tests Used 



1. Goings On at Number Thirty-two (Paddington Bear) 
Deletion pattern 6. 

2. The Puddin 1 Thieves 
Deletion pattern 3. 

3. Christmas Trees 
Deletion pattern 2. 

k. The Boy Who Wouldn't Box 
Deletion pattern 2. 

5. The Claimant 

Deletion pattern 1. 



J- 
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11 
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APPENDIX I 



Number of words in each 'easy 1 subdivision (predictor 
variables) and criterion score for each cloze test. 



Test 



Predictor variables 
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ERIC 





1 

- 


2 


0101 


23.CCC 


30 o 0C0 


0102 


31»CCC 


35o000 


0103 


32.CCC 


36. CCO 


0104 a 


3UCCC 


36.000 


0105 r 


28 o 0CQ 


34o000 


w A V W 


S tc 'J V# Vj 




0107 


25.TCC 


30.000 


0201 


24 CCC 


29.000 


0202" 


3CCCC 


33.000 


0203 


32.CCC 


40.000 


0204 


25oOCO 


31.000 


0205 


33oCCC 


32*000 


0206 1 


26.CC0 


30.000 


0207 


29.CC0 


32.000 


0301 


26.CCC 


30.000 


0302, 


28 lf CCC 


. 36 9 000 


0303 


2e«ccc 


34.000 


0304 


3CoCCC 


36.000 


0305 


35,>0CO 


39.000 


„.0306 


3C«0CC 


35,000 


0307 " 


3U0CC 


37.000 


0401 ' 


24oCCC 


34.000 


0402 


38oCC0 


41.000 


0403 


38oCCC 


41.000 


0404 


35oCCC 


38.000 


0405 


33oCCC 


38.C0O 


0406 


3LCCC 


42.000 


0407 


35oCCC 


42.000 


0501 


3C.CC0 


33.000 


0502 


26,CC? 


27. CCO 


0503 


3 3 o C u C 


32.000 


0504 


26.GCC 


31.000 


0505 . 


33oCCC 


32 o 000 


0506 1 


26.^CCC 


31.CC0 


0507 


3C.CCC 


36.000 


0601 


3? e 0CC 


37,000 


0602 


32«CCC 


37.CC0 


0603 


24-.CCC 


33.000 


0604 




34„CC3 



45.000 
44, 000 
48.000 
48.000 
40 c 000 
45«,000 
43o 000 
45 e 000 
46«600 
45 c 000 
43*000 

44; 000 
370 000 
45.000 

ASftflfifl. 
,47,000 

;a9.qoo 

46*000 
46q000 
460606 

~.49b.0OQ 



26. 000 
31o000 
29. 000 
36.000 

27. 000 
31.000 
23.000 
22 9 000 
31. 000 
30.000 
24. 000 
280 000 
28.000 
3*0.000, 
25. 000 
.3.I0OOO 
32.000 

26*000 
380 000 

.2J«JPQ. 

310000 

.240000 



21.000 
24*000 
21.000 
16.000 
22 o 000 
22.000 
12.000 
17? 000 
20.000 
23.000 
1 80 000 
,26<»000 
17.000 
16,000 
19.000 
2I9OOO 
1 80606 
17,000; 
240 000 
18,000 
2I0OOO 
15*000 



Criterion 
variable. 

5 2, 000 j 
44.000 ■' 
52.000 
50.000 

40*000 ; 

64.000 , 
46.000 
46., 000 
38c 000 
32o000 
58*000 
56*000 
44.000 
40.000 
42.000 
38.000 
44.000 
28.000 
36*000 
,,-40,000 
78.000 ' 
16c 000 



49.000 


37. 000 


26.000 


50.000 


"49. 000 ■ 


T 32.000 


19.000 


26.000 


47,000 


360 000 


23*000 


44,. 000 


48 e 000 


30.000 


18.000 


22 c 000 


50.000 


33*000 


17.000 


68.000 


50.000 


340 000 


20.000 


66.000 


39.000 


30* 000 


19.000 


56.000 


46.000 


30.000 


20.000 


76.000 


50.000 


32*000 


22.000 


50.000 


"45.000 


24*000 


17,000 


48.000 


45 P 000 


33*000 


23o000 


64o000 


44c 000 


31.000 


17.000 


58.000 


47.000 


28*000 


18.000 


54.000 


48irooo 


32*000 


24 o 000 


48c 000 


46 0 000 


31.000 


19.000 


44.000 


44r 000 


22.000 


15.000 


24o000 


47- 000 


32.000 


23.,O00 


44,. 000 


id3 
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0605 

0606 

0607 

0701 

0702 

0703 

0704 

0705 

0706 

0707 i 

0801 

0802* 

ami 

0804 f 
0805- 
0806 
' 0807 
0901 
0902 
0903 . 
0904 
0905 
0906 
0907, 
1001 
1002 1 
1003 
1004_ 
1005! 
1006 1 
1007 

.UP* | 
1102' 

1103 
1104 

li06 

1107. 

1201 

1202 

1203 

1204 

1205 
1206 
1207 
1301 
1302 
1303 
1304 
1305 
1306 



27.CCC" 

32. CCC 
3Co C CC 
3C.CCC 
26eCC0 
27.CCC 
26.0CC 
26- CCO 
35.CCC 
31« vCC 
4UCCC 
4C.CCC 
35.CCC 

35. CC0 
45.CCC 
39.CCC 

36. CCC 
27.CCC 
36cOCC 
26.CCC 

33. GCC 
27eCCC 

32. CCC 
24.0CC 
32*000 

33. CCC 

35. CCC 
32.CCC 

34. CCC 

36. CCC 
35.0CC 

29. CCC 
35oOCO 

35. CCC 

30. CCC 
?8oCC0 
32.GC0 
35oC0C 
33oCCC 
35.CCC 
35.CCC 
40.3CG 

34 0 CCC 

36.0CC' 

32.CCC 

3C.CCC 

34„CCC 

28»CCC 

37. CCC 
3 5c CCC 
39*CCC 



~33.000 
38.000 
37.000 
36eC0O 
32.0C0 
35.000 
3C.0OO 
30.000 
40.000 
34.000 
45.000 
41.000 
44.000 
40.000 
43.GC0 
40.000 
38. COO 
37.CC0 
37.000 
35.000 
37.000 
33.000 
37.CC0 
30.000 
34.000 
32.000 
37.000 
34.000 
36.000 
37.C0O 
39.000 
35.00.0 
39.000 
37.CC0 
38.000 
35.CC0 
35.C00 
33.000 
34.CC0 
4O.C00 
36.000 
41.000 

40.C0C 
40.C00 
38.000 
34.000 
37.CC0 
31.000 
39.000 
36.000 
40.000 
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46T0OO 
45.000 
45.000 
48.000 
43.000 
45.000 
44.000 
47»000 
50.000 
48.000 
50.000 
49» 000 
"50.000 
50.000 
49.000 
50.000 
50.000 
49o 000 
42.000 
46.000 
45.000 

46.000 
46.000 
43.000 
'48.OW 
46.0QO 
47.000 

,.^5,000 
46,000" 

.. 47.000 ;. 
44.000 

45.000 
.45.000 
48.000 

.~AA»JW. 

46.000 

,44.000 
49»000 
48<>000 
47.000 " 
. 49.000,. 

50.000 

r*7?oot r ; 

49o-000 
45.000 " 
47.000 

4^0007 

49.000 
43.000 
50.000 

i«j4 



29. 000 
29. 000 
29.000 
34. 000 
32.000 
32.000 
28.000 
28. 000 
33.000 

30. COO 
44. 000 
42.000 
34.000 
37.000 
41.000 
37. 000 

"36.000 
30.000 
34.000 
„ . 32. 000 
32.000 

31.000 
, 28.000 

35.000 

„3.2«Ppp 
"33.000 
JHUOOG 
30.000 
„3Q*0P0. 
33.000 

:.34»pop; 

33.000 
32.000 
33.000 
,36.000 
33. 000 
33.000 
34. 000 
34.000 

33.000 
33WWO 
35.000 
'29.000 
32.000 
30.000 
36.000 
35.000 
38. 000 



- A 



15.000 
21.000 
19.000 
17.000 
22.000 
21.000 
12.000 
18.000 
22.000 
16.000 
25.000 
29.000 
I8i000 
16.000 
32.000 
24.000 
22.000 
13.000 
20.000 
9.000 
18.000 
J 16.000 
19.000 
20.000 
V.2UO00 
::££*M0p. 
20.000 

17*000 
24.000 

22.0P0: 
16.000 

,22.»P00, 
24.000 

27».0,PP 
19.000 
15.000 
17.000 
20.000 
16.000 
19.000 
22.000 
21.000 

16.000 
21 . 000 

18.000 
15.000 
21.000 
15.000 
19.000 
26.000 
12.000 



32.000 
50.000 
36. 000 
32c 000 
52.000 
44.000 
52.000 
52.000 
40.000 
46.000 
5 80 000 
58.000 
'62.000 
68. 000 
84.000 
56.000 
72.000 
62.000 
58.000 
46.000 
58.000 
58,000 
48.000 
56.000 

; ; ( ¥etoP"o 

36.000 
46.000 
..369 000 
42.000 
;: 32.000 
38.000 
62.000 
60.000 
54»000 
76.000 
56.000 
50.000 
56.000 
60.000 
42.000 
56.000 
42.000 

42.000 
38.000 
40o000 
58.000 
72.000 
72.000 
50.000 
680 000 
72.000 



125. 

\ 



14011 
"1402^ 
1403 

i4d*J 



1501, 
1502; 



"|f70 CO 
32.CC0 
1S.0CG 
33.0CC 

33*CCC 
'36.0GC 
34*000 



37.000 .* 44* OO^^r^Jmn^f: 'i 



37*000 48.000 , 31*000 V 21.000 24*000 



^i5^rrau'oc6 r - 

1505 1 I28*GC0' 



1506 

hot 

1601 
1602 
1603 
1604 
1605 
1606 
1607 



30.0CC 
32*000" 
37*OCO 
33.00C 
34»0CC 
T34*CCC 
,[33* CCO 
<38*0C0 
'32.CC0 



39.000 47*000 m 

36*000 47*000? ••#i31*,000 

rmmmmmmm ^ 

37*000 46*000 
33*000 48*000 



v 23*000 

a 33*000 c 20*000 
16.000 




24*000 



'"'"'^SoOO^^IRr^OlJlf 



36*000 i 

25.000 ?#: 11*000" 14.000 » 
32*000 



21*000 



' mm 



35*000 
37*000 



47*000 
48 o 000 



32*000" 
25*000 



16*000 48*000 



14*000 



30.000 




39*000 
35*000 



1 



48* OO0l^34V600C^ ^2. 066 r~ iCoflo 1 



49* 000: fV ^26* 000 11.000 ' 26.000 



* Predictor variables 

1. Words of 1-4 letters. 

2. Words of 1 syllable. 

3. Words of 1-2 syllables. 

4. Words 'in' common word lists. 

5. Articles, conjunctions, prepositions and pronouns. 
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APPENDIX J 

SAMPLE OF THE DETAILED RESULTS FOR EACH PATTERN FOR 
TWO OF THE SIXTEEN PASSAGES 

(a) The Musical Seal. School 1 Passage 1 
Mean score 24.85 (49.70$) 
Category 1 : Number of letters 

Number of letters/number correct. 



$age correct 
f°age of whole 
%age of correct 



62.71 
67.43 
85.06 



1 Jo 



31.17 
22.00 
13.79 



5.41 
10.57 
1.15 



Excerpt 




1/2 




3/4 




5/6 


7+ 




14 


12 


14 


•7 


1 1 


4 


11 3 


2 


12 


10 


19 


10 


12 


1 


7 1 


3 


6 


4 


26 


16 


12 


4 


6 2 


4 


14 


11 


17 


7 


16 


6 


3 1 


5 


9 


3 


19 


12 


8 


4 


14 1 


6 


10 


8 


20 


16 


9 


4 


11 4 


7 


10 


8 


15 


10 


10 


4 


15 1 




75 


56 


130 




no 

78 


27 


67 13 


%age correct 




74.67 




60.00 




34.62 


19.40 


$age of whole 




21.43 




37.14 




22.29 


19.14 


$age of correct 




32.18 




44.83 




15.52 


7.47 


Category 2: 


Number of syllables 














1 




2 




3+ 




1 


30 


21 


15 


5 


5 


0 




2 


35 


20 


9 


0 


6 


2 




3 


36 


21 


12 


5 


2 


0 




4 


36 


21 


12 


4 


2 


0 




5 


34 


18 


6 


2 


10 


0 




6 


35 


27 


10 


5 


5 


0 




7 


30 


20 


13 


3 


7 


0 




236 


148 


77 


24 


37 


2 





127/ 



Category 3: Common 200 words 







In 




Not 


1 


26 


19 


24 


7 


2 


31 


19 


19 


3 


3 


29 


20 


21 


6 


4 


36 


21 


1 A 


4 


5 


27 


16 


23 


4 


6 


31 


25 


19 


7 


7 


23 


17 


27 


6 




203 


137 


147 


37 


#age 


1 correct 


67.48 




25.18 


$age 


of whole 


58.00 




42.00 


$age 


of correct 


78.74 




21.26 



fa) How Aunt Lettv Killed the Panther. School 1 Passage 4 
Mean score 21.00 (42.OQ06) 



Category 1 Number of letters 

Deletion No No 
Pattern wds Corr 







1/2 




3/4 




5/6 




7+ 


1 


7 


3 


17 


4 


16 


0 


10 


1 


2 


8 


6 


30 


18 


5 


1 


7 


0 


3 


8 


1 


30 


11 


7 


1 


5 


0 


4 


5 


4 


30 


16 


10 


2 


5 


0 


5 


8 


2 


25 


6 


7 


3 


10 


0 


6 


4 


2 


27 


22 


15 


9 


4 


1 


7 


7 


7 


28 


21 


12 


6 


3 


0 




47 


25 


187 


98 


72 


22 


44 


2 



Sfege correct 53.19 52.40 30.55 4.54 



137 



How Aunt Letty Killed the Panther (cont'd.) 
Category 2 . Number of syllables 



1 2 3+ 



1 


34 


7 


15 


1 


1 


0 


2 


41 


25 


8 


0 


1 


0 


3 


41 


12 


8 


1 


1 


0 


4 


38 


19 


9 


3 


3 


0 


5 


38 


9 


10 


2 


2 


0 


6 


42 


28 


8 


6 


0 


0 


7 


42 


31 


8 


3 


0 


0 




276 


131 


86 


16 


8 


0 



#age correct 47.76 26.67 0.00 



Category 5 : In common word lists 



In Not In 



1 


24 


7 


26 


1 


2 


37 


24 


13 


1 


3 


32 


11 


18 


2 


4 


36 


20 


14 


2 


5 


30 


8 


20 


3 


6 


33 


25 


17 


9 


7 


34 


28 


16 


6 


age 


226 


123 


124 


24 


correct 


54.43 




19.36 



Category 4 Part of Speech. 
Prep/Pro 

Conj/Art Other 



1 


15 


6 


35 


2 


2 


26 


20 


24 


5 


3 


19 


5 


31 


8 


4 


23 


12 


27 


10 


5 


18 


6 


32 


5 


6 


17 


14 


33 


20 


7 


20 


18 


30 


16 




138 


81 


212 


66 


•e correct 




58.70 




31.14 



i J<3 



